How to stop RAG hallucinations
Short answer: Most RAG hallucinations are a retrieval failure rather than a model failure: the passage containing the answer was never in the context window, so the model filled the gap from its prior. Fixing them means measuring retrieval separately from generation, then enforcing a citation contract so an unsupported question returns a refusal instead of prose.
If your RAG system returns confident, well-written answers that are simply wrong, you are not alone. It is the single most common reason RAG projects stall before production. The good news: most hallucinations are a retrieval problem, not a model problem, and that makes them an engineering problem you can fix.
Why RAG hallucinates
A retrieval-augmented system does two things: it retrieves context, then it generates an answer from that context. When the answer is wrong, one of two things happened:
- The right context was never retrieved. The model had nothing to ground on, so it filled the gap with a plausible guess.
- The context was retrieved, but the model ignored it or blended it with its own training data.
Teams usually reach for a bigger model to fix this. That rarely helps, because the failure happened before the model ever ran.
Nearly every grounding failure we are called in to fix on enterprise RAG systems traces back to one of those three causes rather than to the model.
The fixes that actually work
1. Fix retrieval first
Naive vector search over badly chunked documents is the usual culprit. We combine semantic search with structured lookups and BM25 (hybrid retrieval) so exact terms, IDs and rare keywords are not lost in embedding space. On one FinTech platform, moving to hybrid retrieval lifted query accuracy by 40%.
2. Enforce a citation contract
The model should only be allowed to answer from retrieved context, and it must return the source for every claim. If the context does not contain the answer, the correct output is “not available in the records,” not a guess. A strict citation contract is how we eliminated “not available” style hallucinations on a medical-records system where wrong answers were unacceptable.
3. Add a knowledge graph where relationships matter
Pure vector search misses relationships between entities. When questions depend on how things connect (who owns what, which entity is in-network), a Neo4j knowledge graph alongside your vector store answers questions a flat index cannot.
4. Measure answer quality
If you are not evaluating retrieval and answer quality, you are guessing. We add evaluation and observability (Langfuse, LangSmith) so regressions show up as numbers, not user complaints.
The failure modes, named
“Hallucination” covers four different faults with four different fixes, and calling them one thing is why teams try the wrong remedy.
Nothing was retrieved and the model answered anyway. The most common and the easiest to fix. The system had no relevant context and generated from parametric memory. The remedy is a relevance floor with a refusal path, not a better prompt.
The right passage was retrieved and contradicted. Rarer than teams assume, and usually a symptom of too much context rather than a defiant model. When forty passages arrive and three are relevant, the signal is diluted.
The passage was retrieved and misread. Frequently a chunking failure: the sentence that qualified the claim was in the adjacent chunk. Structure-aware splitting fixes more of this than any model change.
The source itself was wrong or stale. Not a model failure at all. Your corpus contains a superseded policy and the system faithfully reported it. No amount of grounding work will fix a corpus problem, and this is why freshness is a retrieval concern rather than a content one.
Diagnose which of the four you have before changing anything. Sample twenty wrong answers and label them; the distribution usually surprises people and it tells you where to spend.
Why prompting is not a control
“Only answer using the provided context” is a request. It shifts behaviour and it does not constrain it, and the distinction matters when the consequences are real.
A constraint is structural: the model is never handed content it must not use, and an answer without a supporting passage cannot be returned because the system checks before responding rather than asking the model to police itself.
That is the difference between a demo that behaves and a system that holds. Build the refusal into the retrieval layer, where it is deterministic, rather than into the instructions, where it is probabilistic.
Freshness is a hallucination problem
An answer that was correct last quarter and is wrong today looks identical to an invention, and users report it the same way.
Three mechanisms keep an index honest. Incremental ingestion on change rather than a periodic full rebuild, so the window of staleness is minutes rather than weeks. Deletion that actually removes, because a withdrawn document still in the index is worse than one never added. And recency signals in ranking where your domain has versioned content, so the current policy outranks its predecessor.
Teams that fix grounding and skip freshness get a system that cites its sources accurately and is still wrong, which is a harder problem to explain to a user than an obvious invention.
The takeaway
Before you swap models or add another prompt trick, look at what your retriever is actually returning. Fix retrieval, ground every answer, and measure it. That is how RAG goes from a demo that impresses to a system you can put in front of clinicians, analysts and customers.
Tools referenced
- pgvector
- BM25 / Elasticsearch
- Ragas evaluation
EpochC builds custom RAG development services and refuse to guess. See the graph + vector retrieval case study, the clinical multi-agent case study — zero citation hallucinations in production — or the enterprise RAG guide.