Clinical documentation written
in the practitioner's own voice

A system that captures the practitioner’s voice, transcribes it in several languages and calls an LLMLLM: Large Language Model. An AI trained on huge amounts of text that understands instructions and writes new text, like ChatGPT or Claude. that returns the information structured in different formats. It’s my bachelor’s thesis, built into Streamly, the clinical management platform where I work.

Bachelor's Thesis
UPCUPC: Universitat Politècnica de Catalunya (Polytechnic University of Catalonia). This thesis is from its Manresa school of engineering (EPSEM). · Manresa
Degree
ICT Systems Engineering

From the practitioner’s voice to structured information

Before getting into the details, this is everything the system does. Four steps, where the practitioner has the last word.

  1. 01

    Voice

    The practitioner talks, during the visit or outside it: live through the microphone or by uploading a recording.

  2. 02

    Transcription

    Faster-WhisperOpenAI’s speech recognition model, which turns audio into text. Faster-Whisper is a faster version that runs on our own server. turns it into text on our own server, with the right model for each language. The audio never leaves it.

    caesen

  3. 03

    LLMLLM: Large Language Model. An AI trained on huge amounts of text that understands instructions and writes new text, like ChatGPT or Claude.

    It receives the text with an instruction adapted to the language and the clinic’s specialty. It only extracts what was said; it makes nothing up.

  4. 04

    Structured information

    It comes back in the format the practitioner chooses, who reviews it and saves it. Nothing is stored without their approval.

    Clinical recordSummaryDiagnosisTreatment plan

1,68 sto transcribe one minute of Catalan audio on a GPUGPU: Graphics Processing Unit. The graphics card: it runs thousands of calculations at once, which is why it runs AI models much faster than the regular processor.1,43 son average to generate the record with the LLMLLM: Large Language Model. An AI trained on huge amounts of text that understands instructions and writes new text, like ChatGPT or Claude.

The goal was: to automate a task to give the practitioner time.

No more time spent writing up the visit or the report, during the session or after it, so that time goes to the patient instead. In healthcare, time isn’t just a resource, it’s the direct measure of the quality of care.

Now see it in a real visit

Full recording of the defense: from switching on the module to the clinical note saved during the patient visit.

A problem I didn’t make up

I joined Streamly as an intern, and the thesis came at just the right time. Users kept raising the same complaint: filling in the record for every visit takes time, and that time comes out of the patient’s.

At the same time, the company wanted to bring artificial intelligence into the product, but not as decoration. It wanted it to be useful for something specific. Putting the two together shaped the project.

It was about finding a solution to a real problem.

It took almost six months, from February to July 2026: analysing the problem, building the transcription engine, wiring in the language model, deploying it to the cloud and, at the end, writing it all up in a thesis.

During the session, most practitioners only write down the reason for the visit or mark it as done. Diagnosis, plan and progress are left for later, squeezed in between patients and written from memory. The result: incomplete clinical records.

Streamly visit record without the transcription system’s screenVisit record with the transcription system’s screen alongside: record, upload audio or generate the record with AI
BeforeThe record without the transcription system’s screen, with the clinical fields to be filled in by hand.AfterThe same record with the transcription system’s screen, to fill in the fields by voice.

High-level system flow
Three decisions, all made by the practitioner

I designed it so the AI never does anything on its own. The practitioner turns the module on, decides whether to run the text through the LLMLLM: Large Language Model. An AI trained on huge amounts of text that understands instructions and writes new text, like ChatGPT or Claude. and, whatever happens, reviews the result before saving it. With the module off, the visit is filled in by hand, as always.

Flowchart: turn on the module, transcribe, process with AI or not, choose record or report, and human review before saving
Decision flowAs it appears in the thesis. Every path goes through human review.

Data pipeline
What happens between the voice and the record

Audio comes in two ways: a pre-recorded file or the microphone, live. Both end up as the same transcribed text, and there the practitioner chooses between keeping the transcript as it is or asking the LLMLLM: Large Language Model. An AI trained on huge amounts of text that understands instructions and writes new text, like ChatGPT or Claude. for one of four outputs: summary, diagnosis, treatment plan or clinical record.

Pipeline diagram: file upload or microphone, FFmpeg conversion, Faster-Whisper transcription, transcribed text, language model and the five possible outputs
The full pipelineThe chain of steps data goes through, one after another, until the final result.From the browser to the server and back, with the model’s four outputs at the end.

The details you don’t see

Transcribing while you talk
Live, the browser sends the audio in chunks over a WebSocketA connection that stays open between the browser and the server, so data can flow both ways without asking for it again. and the server transcribes whatever is new every eight seconds. That way the text appears while the practitioner is talking, not when they finish.
Audio that doesn’t know how long it is
The format the browser records (WebMThe audio and video format browsers record in. Here, the audio coming from the microphone.) doesn’t store its duration until the recording ends. To know how many seconds had built up, the server converts the audio and measures it with FFmpegOpen-source program for converting and trimming audio and video. Here it turns the audio into the format Whisper needs and measures how long it is.’s tools every time.
Words cut in half
While testing it I noticed a syllable was sometimes lost right at the cut between two chunks. The fix was to overlap each window with the previous one by half a second. It didn’t come from any theory: it came from using it.
The connection that closed too early
Sometimes the channel closed before the final transcript arrived. I added a check so the browser waits for that last response. I validated it over more than fifty recordings, with a failure rate under 5 %.
Patient visit in Streamly recording audio, with the transcript appearing liveLive transcriptionThe text appears while the practitioner is talking.Patient visit with the clinical record generated by the model from the transcriptThe generated recordOn the left, the transcript and the fields the model proposes, each with a button to move it into the visit.

Every clinic has its own record

A physiotherapist and a podiatrist don’t write down the same things. That’s why the record isn’t hard-coded: the fields come from each clinic’s settings in Streamly, and they are used to build the instruction the LLMLLM: Large Language Model. An AI trained on huge amounts of text that understands instructions and writes new text, like ChatGPT or Claude. receives. If a clinic adds a field, the record includes it without touching anything.

Every instruction carries the same rule: extract only what is said in the transcript, and make nothing up. If a field isn’t mentioned, it stays empty. And the LLMLLM: Large Language Model. An AI trained on huge amounts of text that understands instructions and writes new text, like ChatGPT or Claude.’s temperatureTemperature: the setting that decides how much risk the LLM takes when choosing each word. Low (close to 0): it almost always picks the most likely word and is predictable. High: it takes more risks and is more creative. is low (0.3) so it sticks to the text instead of getting creative.

Which model understands medical Catalan best

To decide, I put together twenty one-minute recordings of real voices, ten in Catalan and ten in Spanish. Each one reads a clinical text written on purpose with the words these models tend to get wrong: terms like cervicalgia or gonarthrosis, anatomical references and patient names. The metric is WERWER: Word Error Rate. The share of words the transcript gets wrong, adds or leaves out compared with what was actually said. The lower, the more faithful.: out of every hundred words, how many come out wrong.

whisper-small mean
29.65%
aina-large-v3-ca median
12.53%
  1. ca_0127.9 %14.0 %
  2. ca_0240.6 %9.4 %
  3. ca_0333.3 %2.8 %
  4. ca_0433.3 %88.9 %hallucination ↓
  5. ca_0522.9 %5.7 %
  6. ca_0622.2 %2.8 %
  7. ca_0725.0 %15.6 %
  8. ca_0854.8 %22.6 %
  9. ca_0913.9 %11.1 %
  10. ca_1022.5 %15.0 %

The error for each sample on the production GPU (Tesla T4). Hover over a sample or tap it to read it on its own.

The Barcelona Supercomputing Center's model goes from nearly one word wrong in three to one in eight. In nine of the ten Catalan samples, AINAA Catalan-specialised speech recognition model from the AINA project at the Barcelona Supercomputing Center. It’s Whisper large-v3 fine-tuned on 710 hours of Catalan audio. makes fewer errors than whisperOpenAI’s speech recognition model, which turns audio into text. Faster-Whisper is a faster version that runs on our own server.-small; it only loses on ca_04. Spanish didn't need it: whisper-small already scores 13.53 % there, so the specialized model is only applied to Catalan.

One of the twenty recordings produced an error above 100 %. It looked like a measurement mistake, but it wasn’t: when it reached a quiet stretch at the end, the model got stuck in a loop and made up text that wasn’t in the audio.

Two decisions follow from this. The figure that gets published is the median, because this single case pushed the mean up to 18.78 %. And professional review became a mandatory step: no automatic transcription system is free from failing on a specific case, however good its average error is.

Generated visit report, open in a text editorEverything arrives as a draftThe generated text opens in an editor: whoever signs the clinical record is the one who validates it.Consent window before recording startsConsent before recordingThe module is switched on per practitioner and asks for explicit confirmation.

Why the system needs a GPU

AINAA Catalan-specialised speech recognition model from the AINA project at the Barcelona Supercomputing Center. It’s Whisper large-v3 fine-tuned on 710 hours of Catalan audio. is more accurate, but also much heavier. The difference between running it on a regular processor or a graphics card decides whether the system can be used during the visit at all.

Transcribing one minute of audio with AINA

AINAA Catalan-specialised speech recognition model from the AINA project at the Barcelona Supercomputing Center. It’s Whisper large-v3 fine-tuned on 710 hours of Catalan audio. on GPUGPU: Graphics Processing Unit. The graphics card: it runs thousands of calculations at once, which is why it runs AI models much faster than the regular processor. 1.68s

Tesla T4An NVIDIA graphics card (GPU) built for servers. It’s the one in the AWS server where transcription runs.. Fast enough to transcribe while the visit is happening.

AINAA Catalan-specialised speech recognition model from the AINA project at the Barcelona Supercomputing Center. It’s Whisper large-v3 fine-tuned on 710 hours of Catalan audio. on CPUCPU: Central Processing Unit. The computer’s regular processor: it does everything, but few calculations at once, which is why AI models run much slower on it than on a GPU. 46.46s

Twenty-nine times slower. Unworkable with the patient in front of you.

Animation sped up ×8. The seconds are the measured ones.

From laptop to cloud

The first version ran entirely on my laptop: the server, transcription, a local LLMLLM: Large Language Model. An AI trained on huge amounts of text that understands instructions and writes new text, like ChatGPT or Claude. and a layer to anonymise the text. It worked, but it was slow and I couldn’t show it to anyone. The final version lives on a GPUGPU: Graphics Processing Unit. The graphics card: it runs thousands of calculations at once, which is why it runs AI models much faster than the regular processor. server on AWSAWS: Amazon Web Services. Amazon’s cloud: it rents out servers and services by the hour, so you don’t need machines of your own., packaged in DockerA tool that packages an application with everything it needs into a “container”, so it runs the same on any server., and only the text goes out to Azure OpenAIMicrosoft’s service that gives access to OpenAI models, such as GPT, from its own servers. Here it’s the LLM that writes the record in production. to generate the record.

Phase 1 architecture: browser and local computer running FastAPI, FFmpeg, Faster-Whisper, Microsoft Presidio and Ollama
Phase 1 · all localThe proof of concept, with PresidioMicrosoft’s open-source tool that looks for personal data in a text (names, phone numbers, ID numbers…) and hides it. for anonymisation and OllamaOpen-source program for running LLMs on your own computer, without sending anything to an outside service. It was used during development. as the local model.
Phase 2 architecture: frontend on Firebase, Docker container on AWS EC2 with FastAPI, FFmpeg and Faster-Whisper, and Azure OpenAI to generate the record
Phase 2 · pre-productionThe audio stays on AWSAWS: Amazon Web Services. Amazon’s cloud: it rents out servers and services by the hour, so you don’t need machines of your own.; only the text reaches AzureMicrosoft’s service that gives access to OpenAI models, such as GPT, from its own servers. Here it’s the LLM that writes the record in production..

One of the decisions that taught me the most was removing something that already worked: the anonymisation layer.

On paper it was a good idea: before sending the text to the LLMLLM: Large Language Model. An AI trained on huge amounts of text that understands instructions and writes new text, like ChatGPT or Claude., Microsoft PresidioMicrosoft’s open-source tool that looks for personal data in a text (names, phone numbers, ID numbers…) and hides it. looked for names, phone numbers or ID numbers and replaced them. But with clinical texts in Catalan it failed both ways. It flagged words like “pàdel” (padel) or “drenatge” (drainage) as personal data, while letting real first and last names through. It damaged the text the LLM received and, on top of that, gave a false sense of security. Protection that only half works can be worse than none, so I removed it and changed the approach: the audio never leaves the infrastructure, and the text travels under contractual guarantees.

The voice never leaves the infrastructure

GDPRGDPR: General Data Protection Regulation. The European law on how personal data can be used; health data gets its strictest protection. classifies health data as a special category, its strictest regime. That shapes the architecture before any model decision: transcription always runs inside controlled infrastructure and, today, the audio never travels to any third party.

ConsultationStreamly
audio
Project infrastructureProject serverAWSAWS: Amazon Web Services. Amazon’s cloud: it rents out servers and services by the hour, so you don’t need machines of your own. EC2EC2: Elastic Compute Cloud. The AWS service for renting virtual servers. This project uses a g4dn.xlarge, with a Tesla T4 GPU. · Faster-WhisperOpenAI’s speech recognition model, which turns audio into text. Faster-Whisper is a faster version that runs on our own server.
the audio stops here
LLMLLM: Large Language Model. An AI trained on huge amounts of text that understands instructions and writes new text, like ChatGPT or Claude.Azure OpenAIMicrosoft’s service that gives access to OpenAI models, such as GPT, from its own servers. Here it’s the LLM that writes the record in production. · Sweden Central

Audio, always local

Faster-WhisperOpenAI’s speech recognition model, which turns audio into text. Faster-Whisper is a faster version that runs on our own server. runs on the same machine that receives the recording. The audio file never leaves the project's infrastructure.

What it costs and where it would go next

Today the expensive piece is the GPUGPU: Graphics Processing Unit. The graphics card: it runs thousands of calculations at once, which is why it runs AI models much faster than the regular processor. server: it is on all day, used or not, and it transcribes recordings one at a time. That is why the thesis's final goals aimed to take that fixed cost out of the way.

476€/mes

Operating cost

GPUGPU: Graphics Processing Unit. The graphics card: it runs thousands of calculations at once, which is why it runs AI models much faster than the regular processor. instance and LLMLLM: Large Language Model. An AI trained on huge amounts of text that understands instructions and writes new text, like ChatGPT or Claude., in a scenario of a thousand daily visits.

0€

Transcription

No per-use cost: the model runs on the project's own infrastructure.

Everything through an API
Running both the transcription and the record through calls to the OpenAI APIAPI: Application Programming Interface. The door through which one program asks another service for something, for example sending audio and getting its text back., with no in-house transcription server. It was ruled out during the thesis because the audio would have to leave the infrastructure; the goal was to find an option that allowed it with the same guarantees the text already has: a European region, a contract and no data retention.
Servers on demand
A KubernetesA system that runs many Docker containers at once: it starts more when there is work, shuts them down when there isn't and restarts one if it crashes.-style infrastructure that starts servers from the DockerA tool that packages an application with everything it needs into a “container”, so it runs the same on any server. image when there is work and shuts them down when there isn't. That way the cost would follow real use, instead of always paying for the busiest moment.

It works. Switching it on is another matter

The system is integrated into Streamly and runs in the pre-production environment. What’s missing before using it with patients isn’t technology.

There are two barriers. The first is cost: the GPUGPU: Graphics Processing Unit. The graphics card: it runs thousands of calculations at once, which is why it runs AI models much faster than the regular processor. transcribes recordings one at a time, so when many practitioners use it at once, paying a bit more isn’t enough; you have to add a whole new server, about €443 more per month each time. The second is legal: working with real patients’ health data requires signing the agreement with the provider, carrying out an impact assessment and defining how consent is requested.

The prototype shows the path is viable. Walking it depends on business decisions, not on code.

And one question is still open: how many minutes it really saves per visit. That needs a study with practitioners using it day to day, and it’s the first thing I would do before rolling it out.

What I take with me

To measure before deciding. Choosing AINAA Catalan-specialised speech recognition model from the AINA project at the Barcelona Supercomputing Center. It’s Whisper large-v3 fine-tuned on 710 hours of Catalan audio. for Catalan wasn’t a hunch: it was twenty recordings and a table.

That removing things is design too. PresidioMicrosoft’s open-source tool that looks for personal data in a text (names, phone numbers, ID numbers…) and hides it. cost me time, and dropping it was one of the best decisions in the project.

And that in healthcare the AI suggests, but a person signs. That’s why everything the system generates arrives as a draft.

What it's built with

Transcription
faster-whisperOpenAI’s speech recognition model, which turns audio into text. Faster-Whisper is a faster version that runs on our own server. small · aina-large-v3-ca from the Barcelona Supercomputing Center
LLMLLM: Large Language Model. An AI trained on huge amounts of text that understands instructions and writes new text, like ChatGPT or Claude.
OllamaOpen-source program for running LLMs on your own computer, without sending anything to an outside service. It was used during development. qwen2.5:3b in development · Azure OpenAIMicrosoft’s service that gives access to OpenAI models, such as GPT, from its own servers. Here it’s the LLM that writes the record in production. gpt-4.1-mini in production
Backend
Python 3.11 · FastAPIA Python library for building web servers: it’s the piece that receives the browser’s requests and answers them. · WebSocketA connection that stays open between the browser and the server, so data can flow both ways without asking for it again. · DockerA tool that packages an application with everything it needs into a “container”, so it runs the same on any server.
Infrastructure
AWSAWS: Amazon Web Services. Amazon’s cloud: it rents out servers and services by the hour, so you don’t need machines of your own. EC2EC2: Elastic Compute Cloud. The AWS service for renting virtual servers. This project uses a g4dn.xlarge, with a Tesla T4 GPU. g4dn.xlarge with Tesla T4An NVIDIA graphics card (GPU) built for servers. It’s the one in the AWS server where transcription runs.
Languages
Catalan · Spanish · English, with automatic detection
Outputs
Transcription · summary · diagnosis · treatment plan · SOAPSOAP: Subjective, Objective, Assessment and Plan. The classic way to structure a clinical note: what the patient reports, what the practitioner observes, what they conclude and what happens next. note

The thesis was submitted in Catalan, so the diagrams on this page keep their text in Catalan.