What You Need to Know
An evaluation metric measures whether a Claude system meets the need that it serves. Each entry on a metric sheet has three parts: the definition, the measurement method, and the target with its baseline. "Accurate" states a goal but gives no measure. "At least 95% field-level extraction accuracy on the held-out set, measured by exact match" is a metric. A held-out set is a set of test examples that nobody used to build or tune the system. Decision-grade metrics are good enough to base a decision on. They are specific enough to measure and complete enough to catch what a single number hides.
This lesson uses one made-up example throughout: a university. It plans a Claude assistant that answers students in a website chat and drafts the weekly student newsletter overnight. The project sponsor's whole brief is "accurate".
Metrics Before the Build
Define metrics before you build the system. The rest of Domain 4 depends on this step. Anthropic's evaluation guidance gives the order: clearly define success criteria, then design evaluations to measure against them. The exam guide's outline for Domain 4 uses the same sequence: define metrics, design datasets, then test.
If you build a system first and measure it later, its standard becomes whatever level it happens to reach. Its metrics are set to match what it already does, and no design decision was tested against the business need. Write the metric sheet before the first prompt, as the university does.
Key Concept
Anthropic's guidance is that success criteria should be specific, measurable, achievable, and relevant. Most use cases need evaluation on several dimensions. The dimensions that the guidance names include task fidelity, consistency, latency, and price.
The Five Metric Families
The guide names five metric families:
| Family | The professional distinction | The single-number mistake |
|---|---|---|
| Accuracy | Match the metric to the task shape: exact match for deterministic outputs, per-field precision and recall for extraction, rubric or judge scores for open-ended generation | One aggregate "percent correct" over heterogeneous tasks hides which capability is failing |
| Latency | Report percentiles, and distinguish time-to-first-token from total completion time, which are different promises to different consumers | A mean that hides the slow responses of the users most likely to leave |
| Cost | Per task at projected volume, the unit-economics framing Domain 1's cost pillar demands | Per-token figures with no volume attached |
| Safety | Refusal correctness in both directions: harmful-output rate and over-refusal of legitimate requests | Counting refusals as if more were always safer |
| Security | Resistance to injection and data leakage, measured under adversarial testing (objective 4.2 designs the cases) | Resistance claimed on paper, never attacked |
The university checks lecture times in chat answers by exact match and newsletter drafts by rubric. A rubric is a written list of scoring criteria. A judge is a model that grades outputs.
Latency targets and alerts belong on tail percentiles such as p95 and p99. The p95 is the time within which 95% of responses finish. This is standard practice for service level objectives (SLOs), because means conceal the poor latency of the users who are most likely to stop using the service. For interactive surfaces, the latency that matters is time-to-first-token, the time until the first part of the response arrives. For pipelines, it is total completion time. Anthropic's latency documentation treats the two as separate measurements. Students in the chat wait for the first token. For the newsletter job, total completion time is what matters.
A safety scorecard with no over-refusal metric lets the system "get safer" by refusing everything. A university assistant that declined every course question would never produce harmful output. If you measure refusal correctness in both directions, over-refusal becomes visible as a failure.
Success Metrics and Guard Metrics
The success metric is what a proposed change should improve. Guard metrics are what the change must not damage. They are typically cost per task, tail latency, and the safety rates. If you declare a success metric without guard metrics, you allow regressions. A variant that improves the success metric but breaches a guard metric has failed. A longer prompt for the university chat that improves accuracy but breaches the p95 guard fails. Objective 4.3's experiments include both kinds of metric, and objective 4.6's production alerts fire on the guard metrics.
Leading Signals, Lagging Proof, and the Proxy Trap
Leading signals are indicators from monitoring that the system's behaviour is changing, such as token spend per request, refusal rate, and retrieval-quality signals. They move before quality visibly drops. Lagging proof is a formal evaluation result. You need both: leading signals for early warning, lagging proof for decisions. A rising refusal rate warns the university. A formal evaluation is the basis for the university's decision.
Any metric sheet risks the proxy trap: a proxy metric that is optimised hard enough stops tracking the outcome it stands for. Response length is a proxy for completeness, and judge score (the score that a grading model gives) is a proxy for satisfaction. Both are useful until the system learns to move the proxy without moving the outcome. Reward longer chat answers, and the university assistant can add filler to them. Audit each proxy periodically. Check it against the real outcome, and stop using it when the two stop moving together.
In the exam
Questions on this objective test whether your metrics are decision-grade. A question that offers one aggregate number, or that proposes to define metrics "once real data exists", is offering you a tempting wrong answer. Two distinctions recur in these questions: latency on tail percentiles, with the latency that matters to the consumer named, and over-refusal measured alongside harmful output.
Exam Traps
Accepting mean latency as the latency metric or service level agreement (SLA) target
Means hide the slowest responses, and the users experiencing the p99 are the ones who stop using the service. Targets and alerts belong on tail percentiles, and the answer also states which latency matters to the consumer: time-to-first-token or total completion.
Deferring metric definition until after launch, when real production data exists
Metrics defined after launch are set to match whatever the system happens to achieve, not to the business requirement. Anthropic's guidance and the order of the exam guide's objectives both put success criteria before the build, and the right answer follows that sequence.
Scoring safety as harmful-output rate alone
A system can get safer by refusing everything. Refusal correctness is measured in both directions: over-refusal of legitimate requests is a failure that has its own metric, not a good result.
Practice Scenario
A team launching a contract-extraction service proposes one metric: overall accuracy of at least 90%, measured on requests they spot-check. The sponsor will sign off only on targets that are fixed before launch, can be re-measured the same way after every release, and show where the service fails. What should the architect do?
Build Exercise
Write the Metric Sheet for One Claude System
What you'll learn
- How to define a metric with a definition, measurement method, baseline, and target
- How to match an accuracy metric to the task shape
- How to state latency as a percentile and name which latency matters to the consumer
- How to separate success metrics from guard metrics
- How to spot the proxies in your own metric sheet before they stop tracking their outcomes
- Pick a production Claude system you know (or design one) and write one metric per family: accuracy, latency, cost, safety, security
Why: The five families come straight from the exam guide, and the sheet forces multidimensional evaluation instead of one aggregate number.
You should see: Five named metrics, one line each — no metric shared between families.
- For each metric record the definition, the measurement method, the baseline, the target, and whether it is a success or a guard metric
Why: A metric without a measurement method cannot be run; a target without a baseline cannot show progress; and every experiment in 4.3 will reuse the success/guard split.
You should see: A five-row table with all five columns filled — no row reading simply accurate or fast.
- For accuracy, match the metric to the task shape and justify it: exact match, per-field precision and recall, or rubric score
Why: Task-shape matching is the recurring accuracy distinction: the wrong grader produces a number that means nothing.
You should see: One sentence naming the task shape and the grader it implies.
- For latency, state the percentile and whether time-to-first-token or total completion time matters to the consumer
Why: These are different targets for different consumers: interactive surfaces depend on the first token, pipelines on the total.
You should see: A percentile target with a named consumer, for example: p95 time-to-first-token for the chat surface.
- Mark each metric as a leading signal or lagging proof, and name one proxy in your sheet that could stop tracking its outcome if optimised hard
Why: Leading signals buy early warning, lagging proof licenses decisions, and every sheet contains at least one proxy waiting to be gamed.
You should see: Each row tagged leading or lagging, plus one sentence naming the proxy and the audit that would show it no longer tracks the outcome.
Sources
- Define success criteria and build evaluations — Anthropic
- Reducing latency — Anthropic
- Mitigate jailbreaks and prompt injections — Anthropic
- Claude Certified Architect Professional Exam Guide (v1.0) — Domain 4 — Anthropic