AI Systems

LLM Observability — Seeing What Happened in Production

Evals and agentic evals tell you what a system will likely do before it ships. Observability tells you what it actually did, for a specific user, on a specific request, right now. The two are complementary and easy to confuse: evals establish confidence before deployment, observability provides ground truth after it.

Quick reference

What it is: Structured tracing, cost/latency dashboards, and drift detection that let you reconstruct exactly what a live system did on a real request.
Mental model: A flight recorder, not a pre-flight checklist — evals are the checklist run before takeoff, observability is what you play back after something goes wrong mid-flight.
Key numbers: Track p50/p95/p99 latency, not just averages — averages hide the tail behavior that actually drives complaints.
Reach for it when: The system is live — this is not optional past a demo, and retrofitting it after an unreproducible incident is far more painful than building it in from day one.

This is the OpenTelemetry model applied to an LLM pipeline — every span carries inputs, outputs, latency, and token counts, so a failure attributes to the exact step that caused it instead of an opaque black box.

Tracing and span-level logging

Every request is captured as a structured trace, following the OpenTelemetry model: a root span for the overall request, with child spans for each retrieval call, each LLM generation call, each tool invocation, and each guardrail check — all tagged with inputs, outputs, latency, and token counts. This lets a failure attribute to the exact step that caused it rather than treating the pipeline as an opaque black box. Purpose-built LLM tracing tools add LLM-aware concepts, like prompt and completion capture, on top of generic distributed tracing.

Token, cost, and latency monitoring

Dashboards tracking token usage, dollar cost, and latency percentiles (p50, p95, p99, since averages hide the tail behavior that actually drives complaints) broken down per route, per model, and per customer segment. This surfaces a specific, common production failure: a single prompt template or a single customer silently becoming disproportionately expensive, often because a chain grew an extra retry loop or a context window quietly ballooned. Rate limiting and budget alerts (per-minute caps, per-tenant spend ceilings) belong in this layer, guarding against a bug or an abusive user generating an unbounded bill.

Drift and anomaly detection

Monitoring output distributions, retrieval quality, and user feedback signals over time catches silent degradation that no single request would reveal. Examples: embedding model drift, where an upstream provider silently updates a model and the vector space shifts underneath a RAG pipeline; a knowledge base going stale as source documents age past a freshness threshold; or a gradual shift in the distribution of user queries that pushes traffic outside the distribution the system was ever evaluated against. Statistical approaches range from simple threshold alerts on rolling eval scores to population-level distribution comparisons between current and baseline traffic.

Productionizing this pattern

What changes past a demo

Observability and agentic evals should share the same trace schema — if the evaluation harness and production tracing log different fields, a production trajectory can't be replayed directly into the eval pipeline when debugging an incident, the single most common integration gap in real systems.
Redact or tokenize sensitive content (PII, credentials, proprietary data) at the tracing layer before it's persisted, not after — full prompt/completion logging is enormously useful for debugging but creates a real compliance surface if handled carelessly.
Alert on leading indicators, not just lagging ones — a rising retry rate or a falling cache hit rate often predicts a cost or quality incident hours before user-facing complaints arrive.
Sampling matters at scale: full tracing on a statistically significant sample plus lightweight metrics-only logging on the remainder, with automatic upgrade to full tracing when an anomaly fires, is the common pattern.
Set SLOs, not just dashboards — an explicit p95 latency target and cost-per-request ceiling that pages an on-call engineer when breached, exactly as for any other backend service.

Real-world applications

Incident response — reconstructing exactly what a customer-reported "wrong answer" incident actually did, step by step, instead of only seeing the final response.
Cost governance — catching a single prompt template or customer silently becoming disproportionately expensive before it shows up as a surprise bill.
Silent quality regression — detecting a knowledge base going stale, or an upstream embedding model changing underneath a RAG system, before user complaints arrive.
Compliance and audit — a full, redacted trace of every production decision is what a post-incident review or regulatory audit actually needs.

Embedded applications

Constrained-device tracing needs the same store-and-forward pattern as any other telemetry — a device can't afford to hold a full OpenTelemetry backend, but it can emit compact, structured spans over MQTT or CoAP to a gateway or cloud collector, exactly the same "batch locally, upload opportunistically" discipline already used for sensor logging.
Drift detection matters more when the model rarely gets updated — a model running on a fielded device might go months between OTA updates (see Kernel Source, Versions & Patching for the same "vendor kernel is years behind mainline" dynamic) — monitoring for input distribution drift is how a team notices the deployed model is quietly falling out of step with reality long before an OTA cycle would otherwise catch it.
Cost monitoring becomes power/bandwidth monitoring — the "denial-of-wallet" risk in the cloud maps directly to denial-of-battery or denial-of-data-plan on an embedded device — a bug that causes an unbounded retry loop is not just an expensive incident, it can be the difference between a device lasting a day or a week on battery.
Full-trace logging is what makes an unreproducible field incident debuggable — exactly as the interview scenario describes for a cloud system, a customer or technician reporting "the device did something wrong" with only a final-state log is undiagnosable — the same root-span-plus-child-spans discipline, sized down for flash-constrained local buffering with periodic upload, is what turns "we have no idea what happened" into an actual root cause.

Gotchas

Field notes

Final-response-only logging is the single most common observability gap — by the time an incident is reported, there's no way to reconstruct which step actually went wrong.
Full prompt/completion tracing without redaction at the logging layer is a compliance liability waiting to be discovered, not a hypothetical one.
Averages hide exactly the tail latency that drives user complaints — a system that looks healthy on p50 can be failing badly on p99, and only percentile tracking catches it.