LLM testing: how to know a change actually helped
Short answer: LLM testing means holding a set of real inputs with known correct outputs, scoring every change against it, and blocking a deploy that regresses. Manual spot-checking feels like testing and is not, because these systems fail in ways that look fine on the three examples you happen to try. The single most valuable thing you can build is the evaluation set, and it should exist before the feature does.
Why ordinary testing does not transfer to LLM testing
A unit test asserts an exact output. A language model produces a different string every run, all of them potentially correct.
So LLM testing scores rather than asserts. Did the answer contain the required facts. Was it grounded in the retrieved context. Did it refuse when it should have. Did the tool call carry the right arguments. Each of those is measurable, and none of them is an equality check.
The second difference is failure mode. Conventional software fails loudly. A model fails plausibly, producing a fluent, well-structured, wrong answer that reads exactly like a right one. Nothing crashes, so nothing alerts.
Build the evaluation set first
Real inputs from your own domain, with agreed correct outputs, held out and version-controlled.
Fifty to two hundred cases is usually enough to start. Fewer than twenty and you are measuring noise. What matters more than count is coverage: the common case, the ambiguous case, the case with no answer in your corpus, the adversarial case, and the ones that have already gone wrong in production.
That last category is the most valuable and the most neglected. Every incident should end with a new case in the set, which is how the suite stops the same failure twice.
Agreeing correct outputs takes longer than building the harness, because it needs a domain expert rather than an engineer. Start it in week one. In our custom RAG development services engagements the evaluation set is the first deliverable, before any retrieval work, because without it every later change is a guess.
Measure retrieval separately from generation
For any system that looks things up, this single split does more than everything else combined.
Retrieval quality. For each question, was the answer-bearing passage actually retrieved. Measured without involving the model at all.
Answer quality. Given correct context, did the model produce a correct, grounded response.
If retrieval sits at 60%, no amount of prompt work on generation will save the system, and knowing that stops you spending weeks in the wrong place. Most disappointing AI is failing at retrieval rather than generation, and that is cheap to establish.
What LLM testing should score
Groundedness. Every claim traceable to retrieved context. The most important measure for anything customer-facing.
Correctness. Against your known answers, scored by exact match where the output is structured and by a model-based judge where it is prose. Validate the judge against human ratings on a sample before trusting it.
Refusal behaviour. When the corpus does not contain the answer, does the system say so. Systems that never refuse are systems that invent.
Format compliance. Where output feeds another system, schema validation is a plain assertion and should be one.
Trajectory, for agents. The right answer via three wrong tool calls is an incident waiting for different inputs.
Cost and latency per case. A change that improves accuracy and triples cost is a trade, not a win, and you should see both numbers together.
Run it as a regression gate
An evaluation suite nobody runs is documentation.
Wire it into continuous integration so every prompt change, model swap, chunking change or retrieval tweak is scored automatically, and a drop below threshold fails the build. That single step converts LLM testing from an activity into a property of the system.
Two practical notes. Pin model versions, or you cannot tell your change from the provider’s. And run the suite on a schedule as well as on commit, because hosted models shift underneath you without a deploy on your side.
Online measurement, after the gate
Offline testing tells you whether a change is safe to ship. It cannot tell you what real users experience.
Track the metrics that predict pain: how often the system refuses, how often users rephrase immediately, how often a conversation escalates to a person, and span-level traces so a bad answer can be replayed rather than argued about. Questions that returned nothing are the most useful log in the system, because they are a direct list of what your corpus is missing.
The takeaway
LLM testing is an evaluation set, a split between retrieval and generation, and a gate in continuous integration. Build the set before the feature, add every incident to it, and score cost alongside quality. Without that you cannot distinguish an improvement from a regression, and these systems are unusually good at appearing to improve while getting worse.
EpochC builds custom RAG development services and AI agent development services with the evaluation harness wired in from the first week. See the graph and vector retrieval case study — +40% query accuracy, measured — or start a project.