AI Observability & LLM Monitoring

What to log per inference

The unit of observability is one inference, and it needs enough context to reconstruct what happened:

The full prompt as sent, after template rendering and context injection — not the template
Retrieved context, with document IDs and relevance scores where retrieval is involved
Model and version, always. Providers update models; without this you cannot distinguish provider drift from your own regression
The raw completion before any post-processing
Token counts in and out, and computed cost
Latency, split between retrieval and generation
User or tenant ID, so you can investigate a specific complaint

Sampling is fine at volume, but log failures and low-confidence outputs at 100%.

Alerts that actually matter

Refusal rate. A sudden rise usually means a prompt change went wrong or the provider tightened safety behavior.

Fallback rate. How often you fall back to a default or escalate to a human. This is your most honest quality proxy in production.

Cost per request, trending. Catches a prompt change that quietly tripled context size. This alone pays for the instrumentation.

Retrieval hit rate. If retrieval starts returning nothing relevant, answers degrade while latency and status codes look perfectly healthy.

p95 latency, separately for retrieval and generation, so you know which half moved.

Output schema validation failures. Where you expect structured output, a rising parse-failure rate is an early warning that model behavior shifted.

Quality drift, which APM cannot see

The distinctive risk: quality degrades with no error, no latency change, and no alert.

Two defenses:

Run your evaluation set on a schedule against production configuration, not just in CI. Score trending down is the signal.

Sample production traffic for human review. A handful of real interactions read by a person each week catches things no metric expresses. It feels unscalable and it's the practice teams most regret skipping.

User feedback as signal

A thumbs up/down is the cheapest quality instrument available, and it correlates with real satisfaction better than most automated scores. Log it against the inference ID so a negative rating retrieves the exact prompt, context, and completion that caused it.

What we set up

Per-inference tracing with prompt, context, model version, tokens, cost, and latency; alerts on refusal, fallback, cost trend, and retrieval hit rate; scheduled evaluation runs against production config; and a feedback signal wired to inference IDs. Built during the engagement, because instrumenting after an incident means the incident is already unexplainable.

FAQ

What should you monitor for an LLM feature in production?

Per-inference traces containing the rendered prompt, retrieved context, model version, token counts, cost, and split latency — plus alerts on refusal rate, fallback rate, cost per request trend, retrieval hit rate, and output schema validation failures. Standard APM catches none of these.

How do you detect AI quality degradation?

Run your evaluation set on a schedule against production configuration rather than only in CI, and sample real traffic for weekly human review. Quality can degrade with no errors, no latency change, and no alert firing, so you need a measurement that looks at content rather than status codes.

Why log the model version on every request?

Because providers update models, and an application that was accurate in March can drift in June with no change on your side. Without the version recorded per inference you cannot tell a provider-side change apart from a regression in your own prompt or retrieval.