Metric Sheet
The artefact written before the first prompt. It holds one metric per family (accuracy, latency, cost, safety, security). Each metric has a definition, a measurement method, a baseline, and a target. Anthropic's guidance puts the order plainly — define success criteria, then design evaluations to measure against them.
Exam context: Options that defer metrics until after launch "when real data exists" reverse the correct order: the target becomes whatever the system happens to achieve. A single aggregate accuracy number is the other common wrong option.
See also: 4.1 Defining Evaluation Metrics
Guard Metric
What a proposed change must not damage, as distinct from the success metric it is meant to improve. Guards are usually cost per task, tail latency, and the safety rates. A variant that improves the main success metric while it damages a guard metric has failed.
Exam context: Every experiment in 4.3 includes both kinds, and production alerts in 4.6 fire on the guards. A change proposal that names only its win condition is incomplete.
See also: 4.1 Defining Evaluation Metrics
Tail Percentile
The p95 or p99 of a latency distribution, where the slow responses that the mean hides actually are. Standard SLO (service level objective) practice puts targets and alert thresholds here, because the users experiencing the tail are the ones who stop using the service.
Exam context: Mean latency offered as the SLA metric or the alert threshold is a common wrong option. The right answer states the percentile and which latency the consumer experiences.
See also: 4.1 Defining Evaluation Metrics
Time to First Token (TTFT)
The interval between sending the prompt and the first token of the response arriving. It is different from baseline latency, which covers processing the prompt and generating the whole response. Anthropic's latency documentation treats them as separate measurements.
Exam context: Users of interactive surfaces notice TTFT. Pipelines depend on total completion time. They are different measurements, and streaming improves only the first of them.
See also: 4.1 Defining Evaluation Metrics
Proxy Trap
The ever-present failure of any stand-in metric: response length as a completeness proxy, judge score as a satisfaction proxy. Optimised hard enough, a proxy stops tracking the outcome it was chosen to represent, and the dashboard keeps improving while the outcome does not.
Exam context: The correct practice is a periodic audit of each proxy against the real outcome. A proxy is dropped when it stops moving with the outcome. Every metric sheet contains at least one proxy that optimisation can exploit.
See also: 4.1 Defining Evaluation Metrics
Case Populations
The four groups that make an eval dataset a set that covers risk rather than a sample. They are representative cases mirroring the production distribution, edge cases at the boundaries where behaviour changes, adversarial cases probing safety and security, and regression cases created from production failures.
Exam context: When a question pairs recurring production incidents with a suite that passes, the missing populations are the answer. Anthropic names the edge cases that teams forget: irrelevant data, overly long inputs, poor user input, ambiguous cases.
See also: 4.2 Designing Evaluation Datasets
Golden Set
The reference evaluation dataset. It is trusted because of three properties. It is versioned, so scores compare across time. It is held out of the prompt, so passing proves capability rather than recall. It is refreshed from production, so it changes as the traffic changes.
Exam context: A static launch set keeps passing while real-world quality falls, because the traffic it tests no longer arrives. Refresh cadence is a design decision, not an accident.
See also: 4.2 Designing Evaluation Datasets
Test-Set Leakage
A hidden way an eval suite becomes invalid: a case that also appears in the prompt as a few-shot example stops being a test, because the model has already seen it. Held out means held out of the prompt, not only out of the training data.
Exam context: Leakage inflates every score the suite reports, so the pass rate rises while capability does not. Near-duplicates leak too, which is why the check belongs in the code that runs the suite rather than in a quick visual read.
See also: 4.2 Designing Evaluation Datasets
Judge Calibration
Validating an LLM judge against human labels on a sample before its scores are used to approve or block anything, and defining in advance what level of agreement is good enough. The judge is a model with its own failure modes, scoring a system it resembles.
Exam context: An uncalibrated judge produces numbers that look certain but prove nothing, however precise the dashboard looks. Anthropic also calls it best practice to use a different model as judge than the one that generated the output.
See also: 4.2 Designing Evaluation Datasets
Statistical Sizing
Sizing and reporting an eval with the statistical methods that non-determinism needs. These are standard errors of the mean and confidence intervals rather than bare scores, clustered standard errors when questions are related, resampled outputs averaged per question, and a power analysis run before the sample size is fixed.
Exam context: Anthropic's statistical evals research notes that clustered standard errors on popular evals can be over three times the naive ones. Growing a dataset to a round number has no statistical reason. Sizing follows risk and variance.
See also: 4.2 Designing Evaluation Datasets
Gate-Then-Validate
The two-stage evidence rule for shipping changes: every candidate must pass the offline eval suite, and an online A/B against the live baseline validates the candidates that pass. The eval set is an approximation of production. Real users are the actual production traffic.
Exam context: A passing suite offered as permission for a full user-facing rollout is a common wrong option. Passing the gate is necessary, never sufficient, for a behaviour change users will feel.
See also: 4.3 A/B Testing and Iterative Improvement
One-Variable Rule
Changing prompt or model or retrieval in a given experiment, never two at once, because a result can be credited to a change only if that change is the only difference. Adopt the first change, establish the new baseline, then test the second against it.
Exam context: "Since we're touching it anyway" is the phrase the exam most often uses to show that changes are bundled. A combined win cannot be credited and a combined loss cannot be traced to its cause, so the next decision rests on an untested belief.
See also: 4.3 A/B Testing and Iterative Improvement
Paired-Difference Test
Comparing two variants on the same questions and analysing the per-question differences, which removes the variance that question difficulty contributes. Anthropic's evals research recommends it for model comparisons as a free variance reduction, since frontier models correlate roughly 0.3 to 0.7 on question scores.
Exam context: The named statistical method behind rejecting a winner chosen by eye. When a question reports a clear improvement on a handful of cases, the right answer asks for sample size and significance.
See also: 4.3 A/B Testing and Iterative Improvement
Staged Exposure
Releasing a risky change in stages rather than all at once. First, shadow runs the new variant on real traffic, with outputs compared offline and no user exposure. Then a canary gives the variant to a small share of live traffic, while the guard metrics are watched and a rollback is ready. Standard deployment practice, taught here at professional level.
Exam context: A rollback plan is not a substitute — shipping to everyone with a revert path still exposes everyone to an unvalidated change. Model-version migrations default to shadow first, because behaviour can shift broadly and silently.
See also: 4.3 A/B Testing and Iterative Improvement
Re-Baselining
Re-measuring every downstream metric after a legitimate change moves the baseline, such as a model-version migration or a corpus refresh that genuinely improved retrieval. Without it, later comparisons run against a baseline that no longer exists.
Exam context: Skipping it makes trends and regressions indistinguishable from the migration's own shift. It is a decision with an owner and evidence, not an automatic step.
See also: 4.3 A/B Testing and Iterative Improvement
Diagnostic Order
The sequence this objective examines: trace and reproduce, isolate the stage the evidence indicates, classify against the failure signatures (the typical symptoms of each failure class), run the cheapest decisive test, fix at the failing layer, and create regression cases so the diagnosis is still useful after the incident.
Exam context: Options that jump to a fix without a diagnosis are wrong options even when the fix might incidentally help. One bad output proves nothing in a non-deterministic system; confirm a pattern across cases.
See also: 4.4 Diagnosing System Issues
Prompt Failure
The class where the instructions are the problem. Signature: format drift, violated output contracts, rules applied inconsistently, and behaviour that changes with phrasing on the same inputs, with failures clustering wherever the instructions leave something unclear.
Exam context: The decisive test is a clarified, structured prompt with everything else held constant. Fix with the cheapest technique. A larger model follows the same unclear instructions, at a higher cost.
See also: 4.4 Diagnosing System Issues
Grounding Failure
Hallucination caused upstream: the context fed to the model was wrong, stale, or missing, and the model answered faithfully from a bad basis. The cue is wrongness correlating with a corpus event (a refresh, a re-import, a migration) or with degraded retrieval-quality signals.
Exam context: The guide's own Domain 4 sample is this case, where the right answer is to investigate retrieval and indexing first. No instruction restores passages the retrieval step no longer supplies, which is why prompt-patching is the wrong option.
See also: 4.4 Diagnosing System Issues
Generation Failure
Hallucination with correct context: the trace shows the right passages present and the answer contradicts or exceeds them. The fix is a set of grounding techniques — allow "I don't know", quote word-for-word first on long documents, require citations and retract unsupported claims, restrict answers to the provided text, verify with chain-of-thought or best-of-N.
Exam context: The grounding/generation split decides where the fix goes: pipeline versus prompt-level discipline. Using one word for both means the fix goes to the wrong layer half the time.
See also: 4.4 Diagnosing System Issues
Model Mismatch
The class where the task exceeds the tier. Signature (the typical symptoms): the hard cases get equally mediocre results across prompt variants. This continues with a clean prompt and verified-correct context. Quality follows task difficulty rather than phrasing.
Exam context: The decisive test is running the failing set on a stronger tier. If the stronger tier meets the target, the upgrade becomes a diagnosed decision that passed the evals. If it does not, the search returns to the earlier stages. Mismatch can go either way. A tier that is more capable than the task needs is the same diagnosis, with the opposite fix.
See also: 4.4 Diagnosing System Issues
Cheapest Decisive Test
The lowest-cost experiment that can confirm or eliminate a hypothesis, run with everything else held constant. Reading existing traces is cheaper than any experiment, so all the evidence you already have is used first.
Exam context: Classification is only half the skill. Questions also ask which test confirms the diagnosis. A test that changes two things at once decides nothing.
See also: 4.4 Diagnosing System Issues
Faithfulness
The measured rate at which output stays within what its supplied context supports. Measuring fabrication as a number, not as single examples, is what makes it possible to prove that a generation-failure fix works.
Exam context: The metric that checks whether the grounding techniques worked. Without it, "we told the model not to invent things" is an intention with no evidence attached.
See also: 4.4 Diagnosing System Issues
Bias Disaggregation
Splitting eval results across user groups and content types instead of reporting one aggregate. A system accurate on average and systematically wrong for one population is failing, and only disaggregated reporting shows it.
Exam context: Detection is Domain 4's job; the governance response (policy, accountability, disclosure) belongs to Domain 5. Questions that mix the two domains are testing whether you know which domain you are being asked about.
See also: 4.4 Diagnosing System Issues
Dominant Term
The largest contributor to a workload's cost or latency, identified from traces and usage data before any optimisation is chosen. If most of the spend is input tokens on one stage, tuning output tokens elsewhere is work that does not reduce the main cost.
Exam context: Optimising the most visible line while the dominant term sits elsewhere is the anti-pattern in this objective that is hardest to notice. Anthropic's Usage and Cost Admin API exists to name the term programmatically.
See also: 4.5 Optimising Tokens, Latency, and Cost
Eval Re-Gate
Re-running the quality suite on the optimised path before an optimisation ships. The target from the metric sheet is a constraint, not a preference, so "cheaper and still passes" is the only win condition.
Exam context: An optimisation that skips its re-gate causes a regression even though the intention was good — the saving is provable and the quality loss is unmeasured. Moving to a smaller model without evals is the standard example.
See also: 4.5 Optimising Tokens, Latency, and Cost
Cost-Performance Frontier
The idea that every optimisation moves along a trade-off curve and does not remove the trade-off. Batch costs turnaround time. Downsizing costs spare capability. Streaming costs nothing and changes only how fast the response seems. The right answer names what is spent and what that saves.
Exam context: An option promising savings with no named cost and no eval evidence is the wrong option. Selection cues matter too: a waiting user rules out batch, and unique inputs rule out caching.
See also: 4.5 Optimising Tokens, Latency, and Cost
Continuous Evaluation
Scoring sampled production traffic automatically between releases: programmatic checks where the property is deterministic, the calibrated judge where it is not. Quality then reads as a trend line against the launch baseline rather than a single result from one point in time.
Exam context: "We eval before every release" is the wrong option, because drift arrives between releases. Anthropic's guidance to automate and prioritise volume is what makes continuous scoring affordable at production scale.
See also: 4.6 Monitoring System Performance
Drift Indicator
A leading production signal that moves before user complaints do. Each one maps to a pipeline stage: token spend per request, refusal rate, tool-failure rate, input-length distribution, retrieval-quality signals.
Exam context: The stage mapping is what makes diagnosis fast when one fires. Wrong options present waiting for concrete complaints as the careful choice, but complaints are in fact the most expensive signal available and the one that arrives last.
See also: 4.6 Monitoring System Performance
Incident Loop
The operational cycle that makes monitoring improve over time: alert, traces, diagnose with the diagnostic order, fix at the failing layer, add the failing cases to the regression cases, and decide whether the baseline legitimately moved.
Exam context: The last two steps are the ones teams skip. Closing an incident without regression cases leaves the fix without a test, so the team works on each bug twice: once to fix it, once to find it again.
See also: 4.6 Monitoring System Performance