We replaced a sprawl of clinical endpoints with one conversational API

The problem

The product had grown a separate endpoint for every clinical capability: one for note generation, one for insurance eligibility, one for searching patient records, one for routing. Each had its own request shape, its own conventions and its own copy of the prompt logic. The client-side code that had to know which endpoint to call for which phrasing had become the most fragile part of the system.

Clinicians do not think in endpoints. A request like “summarise this visit and check whether her plan covers the follow-up” spans two capabilities, and there was no clean way to express it. Adding a fifth capability meant touching the routing logic in every caller.

Underneath that, the records search had a credibility problem. Any answer about a patient that cannot be traced to the document it came from is unusable in a clinical setting, and a plausible-sounding unsourced answer is an active liability.

What we built

1. One entry point, a router behind it

   We built a single conversational endpoint backed by a LangGraph ReAct orchestrator over GPT-4o. The orchestrator reads the clinician’s request and decides which specialised sub-agent handles it — clinical notes, insurance eligibility, records RAG, or routing. Callers stopped needing to know the topology at all.

2. Keep the sub-agents narrow

   Each of the four sub-agents owns exactly one capability, with its own tools and its own tightly scoped prompt. This is the decision that made the system testable: a narrow agent has a small enough behaviour surface to write real assertions against, whereas one prompt attempting all four jobs can only be evaluated by vibes.

3. Ground records retrieval and force citation

   The records agent runs retrieval over pgvector and is constructed so that answers are assembled from retrieved passages with their sources attached. If retrieval returns nothing that supports an answer, the agent says so rather than filling the gap from the model’s prior. That constraint is why the system carries zero citation hallucinations — it is an architectural property, not a prompt instruction we hoped would hold.

4. Give it memory that survives restarts

   Per-user conversation state persists in PostgreSQL rather than process memory, so context survives deploys and the service scales horizontally without pinning a clinician to an instance. Follow-up requests that depend on the previous turn keep working.

5. Delete the duplication

   Consolidating four parallel implementations behind one orchestrator cut roughly 80% of the code in the refactor. Most of what disappeared was prompt logic, request handling and routing that had been copied and lightly edited four times over.

The outcome

• Four specialised sub-agents unified behind a single conversational API.
• 80% of the code removed in the refactor, almost all of it duplication.
• Zero citation hallucinations, because unsupported answers are structurally prevented rather than discouraged.
• Adding a fifth capability became a change in one place instead of a change in every caller.

Why this stack

LangGraph

Explicit graph state and control flow. Agent behaviour you can inspect and test, rather than an opaque chain.

GPT-4o

Reliable tool-calling and low enough latency for interactive clinical use.

pgvector

Vector search alongside the relational data, in one database. No second system to keep consistent.

PostgreSQL

Durable per-user agent memory that survives restarts and scales horizontally.

AWS

Deployment target, with the whole service containerised behind CI/CD.

Frequently asked