What You Need to Know
An evaluation dataset is the set of cases that a system is tested against. A golden set is a dataset that is trusted for comparing scores over time. A grading method scores each case: a code-graded check, LLM-as-judge, or human review. This objective builds the tools that measure the metrics on objective 4.1's metric sheet.
This lesson uses one made-up example throughout: a software company that uses Claude to convert customer bug reports into tickets for its engineers. The company's metric sheet requires tickets that are in the company's format, faithful to the reports and safe from tampering.
Four Case Populations
What a dataset contains decides which risks it covers. Each population protects against a different failure. Hand-written happy-path cases (simple inputs where nothing goes wrong) prove only that the demo works. They never include the inputs that cause failures.
- Representative cases match the mix of inputs seen in production and test the everyday path. They are the bulk of the set. For the company, these are routine reports about sign-in, payment and export problems.
- Edge cases are the inputs at the boundaries where behaviour changes. Anthropic's evaluation guidance names the populations that people forget: irrelevant data, overly long inputs, poor user input, and ambiguous cases. The company adds a report that also contains a complaint about prices, and a very long report. It also adds a report in shorthand and a report that says only "it does not work".
- Adversarial cases are the concrete tests for the safety and security metrics from 4.1. They include injection attempts, leading questions and malformed input. Anthropic's guardrails guidance says to test your workflow with content that deliberately contains injection attempts before deploying. Security is measured under attack. Without attack cases, security is only asserted. The company adds a report that tells Claude to ignore the ticket template.
- Regression cases are made from production failures, so no fixed bug returns unnoticed. A bug with no regression case can return, and then you must fix it a second time. When one ticket named the wrong product, that report became a permanent case.
Golden-Set Discipline
Three properties make a golden set trustworthy.
- Versioned. A score is comparable to last month's only if the set is the same or the change is recorded. Unversioned sets make every trend line meaningless. The company tags each revision of its set.
- Held out. A case that appears in the prompt as a few-shot example (a sample input and output) is no longer a test. Claude has already seen it, so passing proves recall, not capability. Held out means held out of the prompt, not just out of the training data. No example ticket in the company's prompt may appear in its set.
- Refreshed. Production traffic changes. A set that stays fixed after launch slowly comes to test a mix of inputs that the system no longer receives. How often to refresh the set is a design decision to make on purpose. When the company releases a mobile app, reports about the app join the set.
Sizing by Risk and Variance
LLM outputs are non-deterministic. The same report can give a different ticket on each run, so small samples mislead. Anthropic's research on statistical model evaluations describes practices that include:
- Report the standard error of the mean and confidence intervals rather than bare scores. The standard error of the mean estimates how far the average score could move with a different sample of cases.
- Use clustered standard errors when questions are related, because unclustered errors can understate uncertainty substantially. Clustered standard errors treat related questions as one group. Several reports from one customer are related cases.
- Resample non-deterministic answers and average per question.
- Run a power analysis to choose the sample size before running the eval. A power analysis calculates how many cases are needed to detect a difference of a given size.
A round number is not a valid reason to choose a sample size. High-stakes or high-variance behaviours need more cases before a result deserves trust. The company gives reports of lost data more cases than reports of display errors.
Mixed Methodologies: The Pairing Rule
The pairing rule: deterministic checks where possible, a judge where not, and humans to calibrate the judge and audit the ambiguous cases.
- Code-graded checks (exact match, schema validation, regex, programmatic constraints) are cheap, fast and objective. Use them for everything that they can grade. When each case has one correct label from a fixed list, use exact match and not a judge. Anthropic's design principles give the same advice: automate when possible, and prioritise volume. More questions with slightly less informative automated grading are better than fewer hand-graded questions. The company checks every ticket's format with schema validation, a check that the output has the required fields.
- LLM-as-judge scores open-ended quality, such as tone, completeness and faithfulness, at scale. Anthropic describes LLM-as-judge as a grading method for subjective judgements that traditional metrics cannot quantify. Anthropic builds its example judges with explicit scales and definitions. Anthropic calls it best practice to evaluate the output with a different model from the model that generated the output. The judge's rubric has explicit scoring anchors (a written description for each score), and the rubric is versioned like code. The company's judge scores whether each ticket is faithful to the report.
- Human review is the reference standard for grading. Human review is too expensive to use on every case and too important to skip. Use human review to calibrate the judge, to audit samples, and to adjudicate high-stakes or ambiguous cases. At the company, a senior support engineer labels a sample of tickets.
An Uncalibrated Judge Is Not Evidence
The judge is itself a model with failure modes. It scores a system that it resembles. Standard practice is to validate the judge against human labels on a sample before you trust its scores. Until then, the judge's scores are not evidence, however precise they look.
In the exam
The exam tests two design skills. One is to compose a dataset that covers the system's real risks. The other is to pair each metric with a grading method that can honestly score it. "Mixed methodologies" in the exam guide's outline means code-graded checks, LLM-as-judge and human review. Each method is assigned the job that it does best. The right answer follows the pairing rule. When a question describes recurring production incidents alongside a passing eval suite, the missing case populations are usually the answer. Know that the sizing methods exist.
The three golden-set properties are standard evaluation practice, not methods specific to Claude.
Exam Traps
Trusting an LLM judge that nobody has checked against human labels
An unvalidated grader is a model with unknown failure modes scoring a system it resembles. Calibration against a human-labelled sample is what makes its scores count as evidence.
Choosing full human grading as the rigorous option
It wastes human review, the reference standard for grading. Anthropic's guidance is to prioritise volume with automated grading; humans calibrate the judge and audit the ambiguous cases. All-human grading gives less coverage at more cost.
Pasting eval cases into the prompt as few-shot examples, then trusting the pass rate
Claude has already seen those cases, so passing proves recall, not capability. Remove the overlapping cases from the set or the prompt, version the set, and re-score.
Adding more happy-path cases to fix recurring production incidents
More cases from the same population confirm the demo more strongly. The recurring incidents happen in edge, adversarial, and regression cases, which the set never contains. Add those populations rather than more clean variants.
Practice Scenario
A team evaluates a customer-email assistant with 40 hand-written happy-path cases graded by an LLM judge nobody has calibrated. Production incidents keep recurring. What should change?
Build Exercise
Design the Dataset Composition for One System
What you'll learn
- How to compose the four case populations and defend what each protects against
- How to version a golden set and record the source of each regression case
- How to detect few-shot leakage (eval cases that also appear in the prompt as examples) before it invalidates the suite
- How to assign each metric a grading method using the pairing rule
- How to write a judge rubric and plan its calibration against human labels
- For a Claude system you know, write the four populations with counts: representative, edge, adversarial, regression — and one sentence on what each population is protecting against
Why: What the set contains decides which risks it covers. Making the protection explicit exposes the risks that the current set does not cover.
You should see: Four population entries, each with a count and a named risk — not a single undifferentiated case list.
- Record the provenance of every regression case (which incident created it) and the version tag of the current set
Why: Provenance ties each regression case to a real incident, and versioning is what makes scores comparable across time.
You should see: Each regression case linked to an incident, and a version identifier the next eval run will cite.
- Check for leakage: list every few-shot example in the prompt and confirm none appears in the eval set
Why: A leaked case measures memory, not capability, and silently inflates every score the suite reports.
You should see: A leakage check with an explicit result — ideally an automated comparison, not a visual skim.
- Assign each metric from your 4.1 metric sheet a grading method: code-graded, LLM judge with rubric, or human review — and justify each assignment with the pairing rule
Why: The pairing rule is the skill this objective examines: deterministic where possible, judge where not, humans to calibrate and audit.
You should see: Every metric matched to exactly one primary grader, with a one-line justification.
- For the judge: write the rubric with explicit scoring anchors (a written description for each score), version it, and plan the calibration sample of human-labelled cases it must agree with before its scores are trusted
Why: The rubric is the judge specification, and calibration is what makes its scores count as evidence.
You should see: A rubric with a defined scale and anchor descriptions per score, a version tag, and a calibration plan with an agreement threshold.