RAG Consulting: Fixing a System That Exists

Short answer: if your RAG system demos well and disappoints in production, the first question is whether the correct passage is being retrieved at all. That is measurable in about a day, it decides everything downstream, and it is the thing most teams have never checked. A good consulting engagement establishes it before anyone proposes a rebuild.

We do a fair amount of this work, and the outcome is often a list of fixes the client’s own engineers apply. That is a better result than a project.

Start with the diagnosis nobody runs

Take twenty questions the system answered badly. For each, check whether the passage that should have answered it appeared in the retrieved set at all.

Two possible findings, and they point in completely different directions:

The passage was never retrieved. The problem is retrieval — chunking, search strategy, or the embedding model. No prompt change, model upgrade or reranking fixes a retriever that cannot find the passage. Most “hallucination problems” are this, which is why swapping in a bigger model so often changes nothing.

The passage was retrieved and the answer was still wrong. Now it is generation — grounding, prompt, or context assembly. A much smaller and cheaper class of problem.

Teams routinely spend months on the second when they have the first. Running this diagnosis first is the single highest-value hour in any RAG engagement.

That diagnosis is where our own RAG consulting and development work starts, and it is cheap relative to what it saves.

The five things worth auditing

Chunking against your actual document types. Fixed-size splitting is the default in most frameworks and it cuts through tables, separates clauses from their qualifiers, and dilutes embeddings. If a client is still on it, moving to structure-aware splitting usually produces the largest single improvement available. More in RAG chunking strategies.

Whether retrieval is hybrid. Embeddings are excellent at meaning and unreliable at exact tokens. If users search part numbers, error codes, drug names or legal citations and the system only does vector search, a whole class of failure is explained. Adding BM25 alongside is cheap.

The permission model. Does retrieval respect who may see what, or is there one shared index? This is the question that turns a performance review into an incident review, and it is the one most often never asked.

Whether an evaluation set exists. Usually not. Without it, every change is an opinion and the team has been arguing from anecdote. Building one is an afternoon and it changes how the team works permanently — see RAG evaluation.

What happens when retrieval returns nothing. If the system generates anyway, it will confidently invent. Grounding has to be structural — a relevance threshold and a refusal path — not an instruction in a prompt.

What a sensible engagement looks like

Short and specific. A week is usually enough to answer the question that matters.

Days 1–2 — build an evaluation set from real user questions, including ones the system should decline. Measure the baseline.
Day 3 — run the retrieval diagnosis. Establish retrieval failure versus generation failure with numbers rather than opinion.
Days 4–5 — audit chunking, search strategy, permissions and grounding against what the measurement showed.
Output — a ranked list of fixes with expected impact, and the evaluation harness handed over so the team can verify each one.

If the answer is “your chunking is wrong and here is the change”, that is the deliverable. Nobody needs to be sold a rebuild to arrive at it.

When a rebuild is genuinely the answer

Sometimes it is, and the honest signals are structural rather than about quality:

The permission model cannot be retrofitted. A shared index that has to become per-user often reaches back into ingestion far enough that rebuilding is cleaner.
The corpus outgrew the architecture. Something built for a few hundred documents rarely survives a few hundred thousand without redesign.
There is no evaluation harness and no way to add one, because the pipeline is not instrumented and nobody knows what it retrieved.
Per-query economics stopped working at your volume, which is a hosting decision rather than a quality one — covered in RAG as a service vs custom.

Note that “the answers are not good enough” is not on that list. That is a measurement problem until proven otherwise.

Questions to ask a consultant

Turn it around — five questions that tell you whether they have done this before:

“How will you establish whether this is a retrieval or a generation problem?” If there is no clear method, they are guessing.
“What will you measure, and against what?” Anyone not building an evaluation set first is working on vibes.
“What would make you tell us not to rebuild?” A consultant with no such answer has one product.
“How do you handle permission-aware retrieval?” Reveals whether they have shipped anything regulated.
“What do we own at the end?” The evaluation harness in particular — that is the artefact that keeps paying.

The takeaway

Measure before you change anything. The retrieval-versus-generation diagnosis takes a day, costs almost nothing, and determines whether your problem is a two-week fix or a rebuild. Insist on an evaluation set as a deliverable regardless of what else happens — it is the thing that stops the next round of changes being another argument from anecdote.

EpochC provides custom RAG development services and RAG consulting — retrieval-quality measurement, architecture review, and evaluation harnesses you own. See the graph + vector retrieval case study, or start a project.

RAG consulting: what to ask for when your system already exists