LLM observability: what to instrument

Most teams discover they have no observability the first time someone asks a question the system cannot answer: did last week’s prompt change make this better or worse?

There is a transcript, maybe a log line with the final prompt, and no way to tell. The change felt better in manual testing. Nobody can prove it. So it ships, and the next regression is discovered by a customer.

This is the failure mode that separates an LLM feature from an LLM system, and it is not solved by a dashboard. It is solved by deciding what to record and what to measure before you need it.

Logging the prompt is not observability

The common starting point is a log line containing the final prompt and the completion. It is better than nothing and it answers almost none of the questions you will actually have.

It cannot tell you which retrieved chunks were in context, so when the answer is wrong you cannot distinguish a retrieval failure from a generation failure — and those have completely different fixes. It cannot tell you how long each step took, so a slow request is slow for unknown reasons. It cannot tell you what the run cost. And because there is no stable identifier tying the steps of one request together, a system handling concurrent traffic produces interleaved logs that cannot be reassembled.

The unit of observability is not the prompt. It is the trace.

The trace is the unit of debugging

A trace is one end-to-end request, decomposed into nested spans: retrieval, reranking, each model call, each tool invocation, the guardrail check, the final composition. Each span carries timing, inputs, outputs, and metadata. This is exactly the model OpenTelemetry already defines for distributed systems, and LLM systems are distributed systems — the fact that one of the services is a model changes nothing structurally.

Adopting the OTel data model rather than a vendor’s proprietary shape matters more than which tool you point it at. LLM tooling is churning fast; teams that instrumented against a specific vendor’s SDK in the last two years have mostly had to redo it. Emitting standard spans means you can move backends without reinstrumenting, and it means your LLM traces sit in the same system as the rest of your application traces, which is where they belong — a slow request is frequently slow because of the database call the agent made, not the model.

On every span, record the identifiers that let you slice later: model name and version, prompt template version, retrieval configuration, input and output token counts, latency, and a stable trace ID propagated from the entry point. On retrieval spans specifically, record which chunks were returned and their scores. That single field resolves the majority of “why was this answer wrong” investigations, because it immediately answers whether the answer-bearing passage was ever in context.

Version the prompt template explicitly and record the version on the span. Without it, correlating a quality change to a prompt change relies on someone’s memory of when they deployed what.

Evaluation is a test suite, not a dashboard

Tracing tells you what happened on one request. It cannot tell you whether the system is getting better, and this is where most teams stop.

Build a fixed evaluation set of real questions with known-good answers. Sixty to a hundred cases, drawn from actual usage rather than invented, covering the common paths and the known-hard ones. The size matters less than the fact that it is fixed: an evaluation set that changes every time you run it measures nothing.

Evaluate retrieval and generation separately. They fail independently and conflating them wastes enormous amounts of time. Retrieval is measured with recall on the answer-bearing chunk: for each question, was the passage containing the answer actually retrieved, in the top-k the model received? This number is unglamorous, cheap to compute, and in our experience explains more production failures than anything measured downstream. If it is low, no amount of generation work will fix the system.

Generation quality, once retrieval is sound, is measured on faithfulness — is every claim in the answer supported by the retrieved context — and on answer relevance. Ragas implements these reasonably and is a fine starting point. Where a model grades another model’s output, calibrate the judge against human labels on a subset before trusting it, because an uncalibrated LLM judge produces confident numbers with no established relationship to quality.

Include adversarial cases deliberately: questions the corpus genuinely cannot answer. The correct behaviour is refusal, and a system that scores well on answerable questions while confidently inventing answers to unanswerable ones is not a good system. Measuring only the happy path selects for exactly that failure.

Regression gates in CI

An evaluation set that runs when someone remembers to run it is a document, not a gate.

Run it in the pipeline on every change that touches a prompt, a model version, a retrieval parameter or a chunking rule. Fail the build on a regression beyond a defined threshold. This is the mechanism that makes iteration safe, and it is the piece most teams skip.

Set the threshold with the noise floor in mind. LLM outputs vary between runs even at temperature zero, so a naive “no metric may decrease” rule fails builds constantly and gets disabled within a fortnight. Establish the run-to-run variance on your own set first, then set the gate meaningfully outside it. A gate that cries wolf is worse than no gate, because it teaches the team to override it.

Keep the eval fast enough to run on every pull request — parallelise it, cache retrieval where the index has not changed. Once it takes twenty minutes, it moves to nightly, and once it is nightly it stops blocking the change that broke things.

Record the results as a time series. The trend across months tells you whether the system is improving; a single run tells you almost nothing.

Online metrics that predict user pain

Offline evaluation measures the cases you thought of. Production surfaces the ones you did not, and a few online signals are worth more than a wall of charts.

Retrieval-miss rate — the share of requests where no chunk scored above a relevance floor — is the most useful leading indicator we track. It rises before user complaints do, and it rises when the corpus drifts away from what users are asking about, which is the most common slow degradation in a deployed RAG system.

Refusal and fallback rates matter for the same reason. A rising fallback rate means a dependency is degrading. A refusal rate that moves sharply in either direction means either the guardrails tightened or the corpus stopped covering the questions.

Then latency at p95 and p99, per span rather than only end to end, so you can see which stage owns the tail. And explicit user feedback where the product allows it, sampled and reviewed by a human weekly — a small number of read transcripts consistently surfaces failure classes that no automated metric was looking for.

Cost attribution

Token counts on spans give you cost per request, which is the number that lets you answer the two questions leadership actually asks: what does this cost per user, and where would optimisation pay.

Aggregate it by feature, by customer and by prompt version. The distribution is almost always heavily skewed — a small fraction of requests consuming a large share of spend, usually because of long conversation histories or an agent loop that runs longer than expected. That skew is invisible in an average and obvious in a percentile, and once you can see it, cost work becomes targeted instead of speculative. We covered the specific levers in how to reduce LLM costs in production.

Sampling and sensitive data

Recording full inputs and outputs on every span is the right default in development and frequently the wrong one in production — both for storage cost and because those payloads may contain personal or regulated data.

Sample full payloads at a low rate, always record the metadata, and always capture full payloads on error paths, which is where you need them. In regulated environments, redact at the point of emission rather than in the backend: once a payload containing PHI or customer identifiers has left the process, it is in an audit scope you probably did not intend to expand. For healthcare systems this interacts directly with your PHI boundary, which we cover in HIPAA-compliant AI architecture.

What to instrument first

In order of return: trace IDs propagated end to end, so requests can be reassembled at all. Then retrieved chunks and scores on retrieval spans, which resolves most wrong-answer investigations on its own. Then token counts and latency per span. Then the fixed evaluation set. Then the CI gate.

The gate is last because it depends on everything above it, and it is also the one that changes how the team works — once a regression fails a build, prompt changes stop being folklore and start being engineering.

Tools referenced

OpenTelemetry and the GenAI semantic conventions
Ragas — faithfulness, answer relevance, context recall
Langfuse — open-source LLM tracing backend
promptfoo — evaluation harness that runs in CI

EpochC builds the observability, evaluation and CI layer into every RAG system and agent deployment we ship — it is the part that makes the rest maintainable. If you have an LLM system in production and cannot currently answer “did that change help”, book a technical discovery call. We will map what you are recording today against what you would need to gate a release on, and give you the instrumentation plan whether or not you work with us.

LLM observability in production: traces, evals and regression gates