RAG Evaluation: Metrics, Ragas and Golden Sets

Short answer: build a set of a hundred real questions paired with the passage that should answer each, before you build anything else. Without it, every subsequent change is an opinion, and you will spend weeks arguing about whether a tweak helped. With it, you can answer that in minutes.

This is the highest-value artefact in any RAG project and the one most often skipped, because it is unglamorous and it delays the demo.

Separate retrieval failure from generation failure

The single most useful thing evaluation gives you is knowing which half is broken. These need different fixes and get conflated constantly.

Retrieval metrics ask whether the right passage came back at all:

Context recall — was the passage that answers the question in the retrieved set? If this is low, nothing downstream can save you. Fix chunking, add hybrid search, add reranking.
Context precision — of what came back, how much was relevant? Low precision dilutes the prompt and pushes the model toward irrelevant passages.

Generation metrics ask what the model did with what it got:

Faithfulness — is every claim in the answer supported by the retrieved context? This is your hallucination detector, and the metric that matters most in regulated work.
Answer relevancy — does the answer address the question actually asked, rather than an adjacent one?

The diagnostic pattern is simple. High context recall, low faithfulness means retrieval works and the model is inventing — tighten grounding and thresholds. Low context recall means the retriever never found it, and improving the prompt is wasted effort. Most teams that describe “hallucination problems” have a retrieval problem, which we covered in how to stop RAG hallucinations.

Separating those two measurements is the first thing we set up in a custom RAG engagement, because it decides where the rest of the effort goes.

Building the golden set

A hundred question-and-expected-passage pairs. It takes an afternoon and it is the whole foundation.

Use real questions. Support tickets, search logs, the things people actually ask in Slack. Invented questions are cleaner, more grammatical, and unrepresentative — they will make your system look better than it is.

Include the questions that should fail. Things outside your corpus, ambiguous phrasings, questions the assistant should decline. A system that answers everything confidently is worse than one that knows its limits, and you cannot measure that without negative cases.

Record the expected passage, not the expected answer. Answers vary in wording; the passage is objective. This is what makes retrieval measurable independently of generation.

Version it. When the corpus changes, some expected passages move. A stale golden set silently reports regressions that are not real.

Ragas and what it does

Ragas is the common open-source framework here and it computes the four metrics above using an LLM as judge. It is a reasonable default and worth knowing two things about it.

First, LLM-as-judge is itself a model with variance. Ragas metrics are directionally useful and not precise — a move from 0.71 to 0.73 is noise. Treat them as a regression signal, not a score to optimise.

Second, the metrics that need a ground-truth answer (context recall in particular) require you to have written one. The ones that do not — faithfulness, answer relevancy — can run on production traffic, which makes them useful as live monitoring rather than only as a pre-deploy gate.

Alternatives exist and the framework matters far less than having any harness at all.

Run it as a regression gate

Evaluation only pays off if it runs automatically. A harness someone remembers to run by hand is a harness that stops being run in month two.

Wire it into CI. Every change to chunking, retrieval, prompts or the model runs the set and reports the deltas. Fail the build on a meaningful drop in context recall or faithfulness — these are the two that matter — and let the softer metrics inform rather than block.

This is what makes RAG maintainable rather than a system nobody dares touch. It is the same discipline as LLM observability in production, applied before deploy instead of after.

Measure what users experience too

Offline metrics can look healthy while users are unhappy, so pair them with production signals:

Questions where retrieval returned nothing above threshold. This is your content roadmap, ranked by frequency, and it is the most actionable output the system produces.
Answers users explicitly rejected. Low volume, very high signal.
Escalation rate where a handoff exists.
Latency at p95, not the mean — the tail is what people remember, and it grows quietly when you add agentic retrieval.

A workable process

Build the golden set from real questions, including negatives, before writing pipeline code.
Measure the naive baseline. It is usually better than expected, and it tells you what any change must beat.
Change one thing at a time — chunking, then hybrid search, then reranking, then the embedding model. Changing several at once tells you nothing about which helped.
Gate CI on context recall and faithfulness.
Re-baseline when the corpus changes materially.

Step three is where discipline pays. Teams that change four things and see improvement have learned nothing they can build on.

The takeaway

The golden set comes first. Separate retrieval metrics from generation metrics, because they point at different fixes. Treat framework scores as regression signals rather than targets, gate CI on recall and faithfulness, and keep the list of unanswerable questions in front of whoever owns the corpus.

EpochC provides custom RAG development services, including evaluation harnesses you own and can run on every change. See the graph + vector retrieval case study — +40% query accuracy — or start a project.

RAG evaluation: metrics, Ragas and the golden set you need first