What You Need to Know
A trace records one request in full, from input to response. Server software taught us debugging habits, and those habits fail with LLM systems. The same input can produce different outputs. So a single anecdote proves little, and a re-run proves less. A green uptime dashboard shows nothing about whether the answers are right.
This lesson uses one made-up example throughout: a pharmacy chain whose Claude assistant retrieves product pages, calls a stock tool and answers customers. It handles several hundred thousand requests a day. Customers report wrong prices while the uptime dashboard stays green.
Non-Determinism Breaks Anecdote Debugging
Agents make dynamic decisions and behave differently between runs, even on identical prompts. Anthropic's multi-agent research team reported this behaviour in production: users saw agents "not finding obvious information". The engineers could not see why until full production tracing let them diagnose failures systematically.
Re-running a failing input to "see if it happens again" is the anti-method: a method to avoid. Without the trace, neither a reproduction nor a non-reproduction proves anything. A wrong price that is correct on a re-run teaches nothing. Reason about distributions of results, and keep the full context of each request so that any single failure can be explained.
The Trace Is the Unit
A failure you cannot trace is a failure you cannot diagnose. Capture these per request:
- prompt version
- model id
- retrieved context
- tool calls and their results
- token usage
- cost
- latency
- the response
A correlation id is one identifier that every stage records for the same request. It joins the stages of a multi-call pipeline (or the iterations of an agent loop) into one record that you can navigate. A trace that covers each call on its own in a five-stage pipeline cannot tell you which stage failed. At the pharmacy chain, one id joins retrieval, the stock call and the reply.
Suppose the wrong prices began right after a product-page refresh, and the model and the latency did not change. The trace shows what retrieval returned. So start at retrieval, not at the model or the temperature.
Two Tiers at Scale
Nobody reads a million traces. Logging everything forever fails in two ways. It fails on cost. It also fails on privacy, because full prompts and outputs capture user data.
Volume forces a two-tier design. Tier one is cheap aggregate metrics on every request: latency percentiles, token spend, error rates, refusal rates (how often the model declines to answer) and tool-failure rates. Tier two is expensive full payloads on a sample, plus every detected failure. Sampling rates, retention windows and redaction of sensitive fields, such as pharmacy customers' addresses, are architecture decisions that you make at the start. Domain 5 covers the compliance requirements that these decisions serve. This objective covers how to design for them.
Quality Is Not an Infrastructure Metric
Uptime, error rate and latency can all be green while the system confidently produces rubbish, as at the pharmacy chain. A hallucinating system is "up". Quality needs its own instrumented signal: automated evaluation on sampled production traffic. The evaluation scores each sampled answer against a rubric (a written list of scoring criteria) or with an LLM judge (a model that scores the answer). Domain 4's formal evals calibrate the scorer: its scores are checked against the formal eval set. This signal turns "it sometimes makes things up" into a trend that you can measure and set alerts on between formal eval runs. Behavioural monitoring can work at the pattern level without reading individual conversations, and this is how it coexists with privacy constraints.
Green Dashboards, Bad Answers
Complaints alongside healthy service metrics mean quality instrumentation is missing. More uptime alerting adds more of the metrics already green.
Drift Shows in the Cheap Signals
Distribution shifts appear in cheap metrics before quality complaints arrive, because those metrics count every request. Token spend per request slowly increases. Refusal and tool-failure rates rise. Input lengths drift. Alerts on these metrics detect a broken document refresh (like the pharmacy chain's), a prompt regression or changed user behaviour early.
Alert on tail percentiles, not averages. The p99 is the value that only the highest 1% of requests exceed. A stable mean can hide a doubled p99, and the users in that 1% file the tickets.
In the exam
The question says that support reports bad answers while engineering sees green dashboards. The right answer adds instrumentation that would detect the failure. The wrong options add more of the metrics that are already green. "Add more uptime alerting" is a wrong option in nearly every observability question. The guide's Domain 4 sample has the same pattern: confident wrong answers right after a document refresh, while the model and the latency are unchanged. So start at retrieval.
Exam Traps
Adding more uptime and latency alerting when support reports wrong answers
Infrastructure metrics cannot see fabricated answers, so the dashboards stay green while the failure continues. The gap is quality instrumentation: sampled, automated evaluation of what the system actually said.
Debugging a reported failure by re-running the input a few times
Non-determinism makes single re-runs uninformative whether the failure repeats or not. The trace already holds the retrieved context, tool results, and prompt version that explain the failure, so read it instead.
Persisting full prompts and outputs for every request indefinitely
Cost scales with volume and the payloads capture user data, so privacy and retention obligations both fail. The balanced design aggregates everything, samples payloads, and captures every detected failure.
Tracing single calls in a multi-stage pipeline without correlation ids
Without correlation ids the failing stage cannot be found, because five healthy-looking call logs do not reconstruct one broken request. The right answer is the end-to-end trace.
Practice Scenario
A document-assistant serves 200k requests per day across a five-stage pipeline. Support reports that it sometimes makes things up; engineering sees green dashboards (uptime, mean latency). Engineering needs to find where in the pipeline each invented answer starts and how often it happens, without waiting for users to notice. What should the team build?
Build Exercise
Design the Observability Plan for One Pipeline
What you'll learn
- How to define a trace schema that makes any single failure diagnosable
- How to split monitoring into cheap aggregates and sampled payloads
- How to build a calibrated quality signal on production traffic
- How to choose drift alerts on tail percentiles and write a payload privacy policy
- Take a multi-stage Claude pipeline and define its trace schema: prompt version, model id, retrieved context, tool calls and results, tokens, cost, latency, response, correlation id
Why: The trace is the basic record of observability — a failure it cannot explain is a failure the team cannot diagnose.
You should see: A schema listing every field with its source stage, and a correlation id that joins all stages into one record.
- Decide the two tiers: which aggregate metrics run on every request, and what sampling rate plus failure-capture rule gets full traces
Why: Nobody reads a million traces, and logging everything forever fails on cost and privacy at once.
You should see: An aggregates list (latency percentiles, token spend, error, refusal, and tool-failure rates) and a written sampling-plus-failures rule.
- Design the quality signal: what a sampled request is scored against — rubric or judge — and how the scorer is calibrated against formal evals
Why: Quality is not an infrastructure metric; without a scored sample, wrong answers appear as support tickets instead of alerts.
You should see: A scoring definition per sampled request and a calibration note tying the scorer to the formal eval set.
- Choose the drift alerts: thresholds on token spend per request, refusal rate, tool-failure rate, and input-length distribution — on tail percentiles rather than means
Why: Distribution shifts appear in cheap signals before quality complaints arrive, and tail percentiles show problems that averages hide.
You should see: Three or four alert definitions, each naming a signal, a percentile, and a threshold.
- Write the payload policy: what is redacted, how long traces are retained, and which fields never leave the trace store
Why: Full payloads capture user data — sampling, retention, and redaction are design decisions, not problems that an auditor finds later.
You should see: A short policy covering redaction rules, a retention window, and the fields with restricted access.
Sources
- How we built our multi-agent research system — Anthropic
- Usage and Cost API — Anthropic
- Claude Certified Architect Professional Exam Guide (v1.0) — Domain 3 — Anthropic