RAG vs CAG: When to Cache the Whole Corpus

Short answer: CAG loads your entire corpus into the model’s context once, caches the computed state, and answers every query from it — no retrieval step at all. It is dramatically faster and it only works when the corpus is small, stable and shared. RAG stays correct when the corpus is large, changing, per-tenant, or needs citations.

Context windows got long enough that “just put everything in the prompt” became a real architecture rather than a joke. It is worth understanding precisely, because for a narrow class of problem it is clearly better and for everything else it quietly fails.

The mechanical difference

RAG retrieves a handful of relevant chunks per query and puts those in the prompt. CAG puts the whole corpus in the prompt once, caches the model’s computed state over it, and reuses that cache across every subsequent query.

That cache is the entire point. Without it you would re-process the full corpus on every request, which is unaffordable. With it, the expensive prefill happens once and queries become cheap and fast — reported speedups reach 40x on generation time, because you have removed the retrieval hop, the reranking, and most of the prefill.

You have also removed the part of the system that decides what is relevant, which is the trade.

Where CAG wins

Small, stable, shared corpora. A product manual. An internal policy handbook. An API reference. Something that fits comfortably in context, changes on a release cadence rather than hourly, and is the same for every user.

Latency-critical paths. Removing retrieval removes a network hop, a vector search and usually a rerank. If you are fighting for sub-second responses, that is meaningful.

Questions that span the corpus. This is the underrated one. “What is inconsistent between these policies?” is genuinely hard for RAG, because retrieval returns the top-k passages and the answer requires seeing everything. CAG has everything already.

Corpus size and churn decide this, and both are things we measure before proposing an architecture in RAG development work.

Where CAG breaks

Size. The corpus must fit the context window. Long windows have made this less restrictive, but “fits” is not the same as “should” — you pay to prefill it and, in cached form, to hold it.

Freshness. Any change to the underlying documents invalidates the cache and requires rebuilding it. For a handbook updated monthly, fine. For a ticketing system updated continuously, unworkable.

Per-tenant data. This is the constraint that eliminates CAG most often in practice and the one teams notice last. If different users may see different documents, a single shared cache is a data leak by construction. You would need a cache per permission set, which destroys the economics. Anything with per-user permissions stays on permission-aware retrieval.

Citations and auditability. RAG knows which passages it used, because it retrieved them. CAG saw everything, so attributing a claim to a source is a harder problem. In regulated work where every claim needs provenance, that matters.

The hybrid most production systems land on

The framing as a binary is wrong. What works is routing.

Keep a CAG path for the hot, shared, stable core — the handbook everyone asks about — and a RAG path for the long tail, the fresh data and anything permission-scoped. Route per query, and evaluate both paths against the same golden set so you can see when the router sends a query to the wrong one.

This is more moving parts, so only build it when you have measured that one path alone is insufficient. Most teams do not need it.

Cost behaves differently

RAG cost scales with query volume: each query retrieves and prefills a few thousand tokens. CAG front-loads a large prefill, then queries are cheap against the cache.

That means CAG gets cheaper the more queries you serve against a stable corpus, and worse the more often the corpus changes, because each change re-pays the prefill. High query volume plus low change rate favours CAG. The opposite favours RAG — and if your corpus changes hourly, CAG’s economics never recover.

Do the arithmetic on your own numbers before believing anyone’s benchmark.

How to decide

Seven questions, in order of how often they settle it:

Is the corpus per-tenant or permission-scoped? If yes, RAG. Stop here.
Does it change more than daily? If yes, RAG.
Does it fit in context with room for the conversation? If no, RAG.
Do you need per-claim citations? If yes, RAG is far easier.
Are queries dominated by corpus-spanning questions? If yes, CAG is genuinely better.
Is latency the binding constraint? CAG helps.
Is query volume high against a stable corpus? CAG’s economics improve.

Questions one and two eliminate CAG for most business use cases, which is why RAG remains the default rather than a legacy choice.

The takeaway

CAG is a real technique with a narrow, well-defined fit: small, stable, shared corpora where latency matters and citations do not. Check the permissions and freshness questions first — they decide it faster than any benchmark. For most custom RAG development, retrieval stays the right architecture, and the interesting work is making retrieval better rather than removing it.

Sources

EpochC provides custom RAG development services — hybrid retrieval, permission-aware indexes and evaluation harnesses. See the graph + vector retrieval case study, or start a project.

RAG vs CAG: when to stop retrieving and cache the whole corpus