We built an AI scribe that turns a consultation into completed clinical forms

The problem

Clinicians were losing hours every day to documentation. The consultation itself was the short part; what followed was transcribing it and then re-entering the same facts across more than ten mandatory forms, each wanting a different subset in a different shape.

Off-the-shelf transcription solves only the first step. It produces a wall of text, and the actual work — deciding which utterance belongs in which field of which form — remains entirely manual. A transcript is not documentation.

Consultation audio is also genuinely difficult input: two speakers who interrupt each other, clinical terminology and drug names that general speech models mangle, and ambient noise. And the accuracy bar is high, because a documentation error in a medical record has consequences well beyond an inconvenience.

What we built

1. Transcribe with a model that handles real consultations

   Deepgram Nova-2 handles speech-to-text in real time, with speaker diarisation so the clinician’s speech is distinguishable from the patient’s. That separation matters downstream: “I have been taking ibuprofen” means something entirely different depending on who said it.

2. Extract structured facts, not prose

   A LangGraph agent works over the diarised transcript to pull out the clinical facts as structured data — presenting complaint, history, medications, observations, plan — rather than producing a summary paragraph. Summaries have to be read and re-entered; structured facts can be routed into fields automatically.

3. Map facts onto every form that needs them

   Those extracted facts populate more than ten mandatory forms. One fact reaches every form that asks for it, which is where the real time saving comes from — the clinician was previously re-entering the same information repeatedly, and that duplication disappears entirely.

4. Score every field, and make review the default

   Each populated field carries its own confidence score, surfaced in the review interface. Uncertain fields are flagged for attention instead of being quietly filled. The scribe drafts and the clinician signs off — we designed it so the system never writes to a record unreviewed, because in clinical documentation a confident wrong answer is worse than a blank field.

5. Ship it behind an API

   FastAPI serves the pipeline, so the scribe integrates with the existing clinical workflow rather than becoming another application clinicians have to context-switch into.

The outcome

• 10+ mandatory forms auto-filled from a single consultation.
• Real-time speech-to-text, so documentation drafts exist by the time the consultation ends.
• Per-field confidence scoring, making clinician review fast and targeted rather than a full re-read.
• Hours of documentation removed per provider, per day — the duplicate data entry across forms disappears entirely.

Why this stack

Deepgram Nova-2

Real-time transcription with diarisation that holds up on overlapping speech and clinical vocabulary.

LangGraph

Structured extraction as an inspectable graph, so a mis-extraction is traceable to a step.

OpenAI

Clinical-language understanding for mapping utterances onto the right structured fields.

FastAPI

Fits the scribe into the existing clinical workflow instead of adding another application.

Frequently asked