Aliases: LLM observability, LLM tracing
AI Observability
Tracing, logging, and monitoring for LLM systems — seeing what the model was asked, what it did, what it cost, and where it failed.
What is AI observability?
AI observability extends classic monitoring to LLM behavior: capturing full traces of each interaction — prompts, retrieved context, tool calls and results, model outputs, token counts, latency, cost — so you can answer “why did the AI do that?” after the fact. Tools include Langfuse, LangSmith, Braintrust, Arize Phoenix, and OpenTelemetry’s GenAI conventions for teams extending existing observability stacks.
Why normal APM isn’t enough
Your existing dashboards say the request returned 200 in 3 seconds. They can’t say the answer was wrong, the retrieval pulled irrelevant chunks, the agent looped through 14 unnecessary tool calls, or this customer’s queries cost 40× the median. LLM failures are semantic, invisible to status codes — observability for AI means logging content, not just metadata.
What to capture (the practical checklist)
Per request: the exact rendered prompt (post-template), model + parameters, retrieved documents with scores, every tool call and result, output, token usage, cost, latency breakdown (TTFT vs total), and user feedback signals (thumbs, retries, rage-refresh). Per agent task: the full trajectory as one trace with spans per step. This corpus is doubly valuable — it’s also the raw material for your eval sets and distillation data.
The flywheel
Observability and evals form the production loop: traces surface failures → failures become eval cases → evals gate the fix → monitoring confirms it in prod. Teams that skip tracing end up debugging by prompt-tweaking blind — changing things and vibing whether it helped.
What people get wrong
- Logging prompts but not context. When RAG misbehaves, you need what was retrieved, not just the question.
- Ignoring privacy in traces. Full-content logging captures PII by design; apply redaction, retention limits, and access controls like any sensitive datastore.
- Cost dashboards without attribution. A total spend number is useless; break down by feature, model, and customer to find the 5% of traffic burning 60% of budget.
Historical figures and technical concepts for informational purposes only. Not technical, professional, legal, or financial advice. Sources: Official Documentation.