What You Need to Know
Launch-time evals prove readiness once. Monitoring proves that readiness stays true in production. Drift is a change after launch that moves quality. A baseline is the measured level of a metric. A guard metric is a metric that a change must not damage. This objective is the last step of the Domain 4 cycle. It uses the 4.1 metrics, the 4.2 graders and the 4.4 diagnosis in production. Every incident adds cases to the dataset.
This lesson uses one made-up example throughout: an accountancy firm whose Claude assistant answers client questions from its tax guidance library. Since launch, the firm has refreshed the library, edited the prompt and added a client portal.
The Quality Practice on Top of Domain 3
Domain 3 built the infrastructure: what a trace captures, how sampling works, and how aggregate metrics are collected. Domain 4 runs the quality practice on that infrastructure: what dashboards show, and what happens when a line on a dashboard moves. The dashboards track the 4.1 metric sheet (the list of metrics and targets from lesson 4.1) against its targets, not only infrastructure health. The metric sheet includes accuracy signals, latency percentiles, cost per task, and refusal and error rates.
Uptime does not measure wrongness. A green uptime dashboard proves that the service is up. It does not prove that the service is right. A system can be fully up, fast on average and confidently incorrect. So quality monitoring is a separate discipline. It has its own scored signals. The firm's assistant can answer quickly and still quote last year's guidance.
The Domain 3/4 Boundary
Trace capture and sampling design belong to Domain 3's infrastructure. Which quality signals to score, alert on and act on belongs here.
Continuous Evaluation
Formal eval runs give proof that comes late, at each release. Drift arrives between those runs. Continuous evaluation automatically scores a sample of production traffic. Programmatic checks score deterministic properties. The calibrated judge from 4.2 (an LLM that scores outputs and was checked against human labels) scores the rest. Quality becomes a trend line against the launch baseline, not a single result from one point in time. At the firm, a check confirms that each answer cites a guidance note. The judge scores the advice.
Anthropic's evaluation guidance says to automate when possible and to prioritise volume with automated grading. That guidance makes continuous evaluation affordable at production scale. The Usage and Cost API documentation lists these use cases: monitoring product performance, and measuring whether system changes improved it. The guardrails guidance adds security: continuously monitor outputs for signs of successful injection.
Drift and Leading Indicators
Input distributions shift, corpora refresh, models migrate and prompts accumulate edits. These changes follow no release schedule. Leading indicators move before user complaints do. Each leading indicator maps to a pipeline stage. So when an alert fires, finding the failing stage with the 4.4 method is fast.
- Token spend per request: inputs grew, or a prompt edit changed behaviour
- Refusal rate: traffic shifted, or a policy prompt interacts badly with new inputs
- Tool-failure rate: an upstream API changed
- Input-length distribution: a new client integrated
- Retrieval-quality signals: a corpus refreshed
At the firm, the library refresh maps to retrieval-quality signals. The new portal maps to input-length distribution.
Waiting for concrete user complaints is the expensive alternative. Complaints are the signal that arrives last. Each complaint comes from a user whose problem the leading indicators could have caught earlier.
Alerts, Baselines, and the Incident Loop
Alerts fire on guard metrics at tail percentiles. Their thresholds come from measured baselines. Averages hide the problems, so alerts on averages fire late or never. Alerts on everything send the team so many alarms that it ignores the real alert. To keep the real alerts visible, the team deliberately chooses what not to alert on. The firm sets tail thresholds from its baselines. It also names the signals that do not alert anyone.
When an alert fires, the incident loop runs:
- Alert
- Traces
- Diagnose, in 4.4's order
- Fix at the failing layer
- Create regression cases (4.2)
- Re-baseline, if the baseline legitimately moved
The last two steps make monitoring improve over time. If the team skips the regression cases, the same failure returns under a new ticket number. If the team skips the re-baseline after a legitimate change, every future comparison is meaningless, because it uses an old baseline that is no longer correct. A legitimate change is, for example, a model migration or a corpus refresh that genuinely improved things. When the firm traces wrong answers to the refreshed library, it fixes retrieval, adds regression cases, and re-baselines if the refresh improved things.
In the exam
The exam tests how to run this practice. It also tests where Domain 3 ends and Domain 4 begins. Questions test the difference between uptime and wrongness directly. When a question asks "what should the team monitor?", check whether it wants instrumentation (Domain 3) or practice (Domain 4). The sample scenario in the guide is a corpus refresh that shows as confident wrong answers.
Exam Traps
Presenting green uptime and mean-latency dashboards as evidence the system is healthy
Uptime does not measure wrongness. A system can be fully up, fast on average and confidently incorrect. Quality needs its own scored signals, tracked against the 4.1 targets.
Skipping continuous evaluation because the team evals before every release
Drift arrives between releases: corpora refresh, traffic shifts, and prompts accumulate edits on no release schedule. Launch evals measure quality at launch time. Only continuous scoring of sampled traffic measures quality now.
Closing incidents without creating regression cases from the failures
No test protects the fix: nothing detects the same failure returning, so it comes back under a new ticket number. Create regression cases (4.2) from each failure so the next run catches it.
Practice Scenario
Three months after launch, a procurement assistant's satisfaction scores fall. Release evals all passed; the uptime and mean-latency dashboards are green. The team must learn what has changed before altering the system, and catch any further decline before users report it. What should the team do?
Build Exercise
Write the Monitoring Page for One System
What you'll learn
- How to choose what gets continuously scored on sampled production traffic and with which grader
- How to select drift indicators and map each to the pipeline stage it indicates
- How to set alert thresholds on guard metrics at tail percentiles from measured baselines
- How to write the incident loop as an operational runbook
- How to test a monitoring design against the on-call engineer question
- For a Claude system you know, list what is continuously scored on sampled production traffic, and with which grader (programmatic check or the calibrated judge from your 4.2 exercise)
Why: Continuous evaluation is the practice that turns quality from a single result at release time into a trend line.
You should see: A short list of scored signals, each paired with its grader and sampling approach.
- Choose the drift indicators you will watch (token spend per request, refusal rate, tool-failure rate, input-length distribution, retrieval-quality signals) and map each to the pipeline stage it indicates
Why: Leading indicators move before complaints, and the stage mapping is what makes diagnosis fast when one fires.
You should see: Each chosen indicator with the stage it indicates, for example: retrieval-quality signals indicate a problem in the index and embedding pipeline.
- Set alert thresholds: which guard metrics, at which tail percentiles, derived from which measured baselines — and name one thing you deliberately will NOT alert on, to keep the real alerts visible
Why: Alerts on averages fire late; alerts on everything make the team ignore them over time. Both failures are design choices.
You should see: A threshold table (guard metric, percentile, baseline, threshold) plus one explicit non-alert with its justification.
- Write the incident loop as a runbook: alert, traces, diagnose, fix, regression cases, re-baseline decision
Why: The loop is what makes monitoring improve over time. The team follows it during an incident only if it was written down before the incident.
You should see: Six runbook steps with owners, ending in two explicit questions: which cases get added to the eval set, and did the baseline legitimately move.
- Test the page with the 2am question: when this system fails three months from now, what will the on-call engineer see, and how fast does it lead them to the failing stage?
Why: A monitoring design is good only to the extent that someone who did not build the system can follow the path from signal to stage.
You should see: A walkthrough from a plausible failure to the alert that fires, the dashboard line that moved, and the stage the indicator indicates.
Sources
- Usage and Cost API — Anthropic
- Define success criteria and build evaluations — Anthropic
- Mitigate jailbreaks and prompt injections — Anthropic
- Claude Certified Architect Professional Exam Guide (v1.0) — Domain 4 — Anthropic