Enterprise RAG Architecture: What Changes at Scale

A RAG demo over a folder of PDFs and an enterprise RAG deployment share almost no engineering. The demo needs a vector store and a prompt. The deployment inherits permissions, document churn, multiple source systems, formats that resist clean extraction, and users who will notice immediately when it is wrong.

This covers what actually changes at enterprise scale.

Most RAG failures are retrieval failures

When a RAG system returns a wrong answer, the instinct is to blame the model and reach for a bigger one. In our experience retrieval is at fault far more often: the passage containing the answer was never in the context window, so the model filled the gap from its prior. No model upgrade fixes that.

Audit retrieval in isolation first. For a representative set of real questions, is the answer-bearing passage actually being retrieved? That single measurement usually reframes the entire problem and points at chunking, embeddings or query construction rather than generation.

Entitlement-aware retrieval

This is the requirement that most cleanly separates enterprise from demo. Different users may see different documents, so retrieval must filter by entitlement before it ranks.

Filtering after retrieval is a leak waiting to happen — the vectors were still searched, the results were still assembled, and one bug in the post-filter exposes documents to someone who should never have seen them. We attach permission metadata at ingestion and enforce it inside the retrieval query using the requesting user’s identity. It is a security boundary, not a filtering nicety.

An internal knowledge assistant that surfaces the wrong HR document to the wrong employee is an incident, not a bad answer.

Getting entitlement inside the retrieval query rather than bolted on afterwards is the single change we make most often in custom RAG development on systems that already exist.

Chunking decides your ceiling

Chunk boundaries determine what can be retrieved at all:

Split mid-table and the header separates from its rows
Split on fixed token counts and a definition separates from the term it defines
Chunk too large and the embedding averages several topics into a vector that matches nothing precisely

Chunk on document structure rather than character counts, preserve the context each chunk needs to stand alone, and keep metadata attached so retrieval can filter before it ranks. Unglamorous work that moves accuracy more than almost anything downstream.

Hybrid retrieval, because embeddings miss exact terms

Vector search finds text that means something similar. It is reliably weak on exact identifiers — part numbers, error codes, statute references, proper nouns — because those carry little semantic signal. A user searching an exact code frequently gets topically related passages that do not contain it.

Pairing semantic search with BM25 keyword matching covers both. Adding a knowledge graph covers a third case: questions about how entities relate, which embeddings structurally cannot answer because the answer lives in no single passage.

On a FinTech retrieval platform, combining Neo4j graph traversal with vector search lifted query accuracy 40% and answer relevance 35% against a vector-only baseline, served end to end in under 200ms with Redis caching the hot paths.

When a graph is worth the complexity

Audit your failed queries. If users ask “what does this document say about X”, vector search suffices and a graph is overhead. If they ask “which suppliers connect to this counterparty through a subsidiary”, vector search cannot answer it at all.

The insight is not that graphs beat vectors — it is that they fail on different questions.

Choosing a vector database

The choice matters less than teams expect, and the wrong framing is expensive. pgvector is the right default when your data already lives in Postgres: one datastore, transactional consistency, no synchronisation problem.

Dedicated vector databases earn their place at real scale or when you need index types Postgres lacks. The costly mistake is adding a second datastore purely for embeddings and inheriting a sync problem — documents update, embeddings go stale, and retrieval quietly degrades against data that no longer matches the source.

Keeping the index fresh

If ingestion is a one-off script someone ran during the build, retrieval degrades continuously against a corpus that no longer resembles reality. Nobody notices until a user gets a confidently outdated answer.

Build incremental ingestion with change detection: updates re-embed only what changed, deletions remove their vectors, and the index tracks its source. Full reindexing becomes a recovery procedure rather than routine operation.

Grounding is architecture, not instruction

Asking a model in the prompt not to invent sources is not a control.

Systems that reliably avoid hallucinated citations are built so answers are assembled from retrieved passages with sources attached, and when retrieval surfaces nothing supporting, the system says so rather than generating prose. On a clinical orchestrator we built, that constraint is why the records agent carries zero citation hallucinations — a structural property of how answers are composed, not a hopeful instruction.

Evaluation you can run on every change

Without an evaluation set you cannot distinguish improvement from regression, and RAG systems are unusually good at appearing to improve while getting worse on cases you stopped checking.

Build a question set with known correct sources early, and measure retrieval quality separately from answer quality. Separating the two makes debugging tractable: if retrieval sits at 60%, no amount of prompt work on generation will save the system, and knowing that stops you spending weeks in the wrong place.

Cost at scale

Embedding a large corpus once is cheap. Re-embedding it repeatedly because ingestion has no change detection is not. Neither is retrieving fifty chunks per query when eight would do, then paying for all of them in the context window on every request.

Retrieval breadth is a cost lever as well as an accuracy one, and it should be tuned against your evaluation set rather than guessed.

The takeaway

Enterprise RAG is a retrieval engineering problem with a language model attached, not the reverse. Get entitlement enforcement, chunking, hybrid search, incremental ingestion and evaluation right, and the model choice becomes almost incidental.

Tools referenced

EpochC builds RAG and retrieval systems and AI agents. See the graph + vector retrieval case study — +40% query accuracy, sub-200ms — the clinical multi-agent case study, or start a project.

Enterprise RAG architecture: permissions, freshness and evaluation