What You Need to Know
An offline eval tests a candidate change on a fixed dataset before users see it. An online A/B test splits live traffic between the current baseline and the candidate. A guard metric must not get worse. Together with the 4.1 metrics and the 4.2 datasets, offline evals, A/B tests and guard metrics form a loop. The loop converts a hypothesis into a production change that is attributable, statistically valid and reversible.
This lesson uses one made-up example throughout: a payments company that uses Claude to summarise each disputed payment for its review staff. The team wants a rewritten summary prompt and a transaction-history lookup.
The Gate and the Validation
Offline evals are the gate, a check that a candidate must pass. Online experiments are the validation. The offline suite is quick and repeatable enough to run on every candidate. Failing it stops the candidate. A pass is necessary but not sufficient for a change to user-facing behaviour. The eval set approximates production, and production means real users. The passing candidate proves its real-world impact in an A/B test against the live baseline, on real metrics and traffic. The company tests its new prompt on past disputes, then on live disputes.
Anthropic's model-selection guidance sets the same order for model changes:
- Build benchmark tests specific to your use case.
- Test with your actual prompts and data.
- Weigh the performance and cost trade-offs.
Passing the offline evals does not replace the online validation, and a rollback plan does not replace either of them.
One Variable at a Time
Only an isolated change can be credited with a result. If the prompt is rewritten and the model is upgraded together, a win cannot be credited. Also, the causes of a loss cannot be separated. The next decision then depends on an unproven belief, not a finding. So release the first change, establish the new baseline, then test the second change against that baseline. The company tests its new prompt alone. Then it tests the lookup against that baseline.
This feels slower and costs less. Wins with no known cause get reverted by accident, and losses with no known cause get debugged without evidence. Each extra change added with the reason "Since we're touching it anyway" removes the attribution. The test then cannot show which change caused the result.
The Evidence Bar
Non-deterministic outputs make measurements noisy. So any result needs more evidence before you declare it. Anthropic's statistical approach to evals supplies the methods:
- Report the standard error of the mean and confidence intervals, not bare scores. The standard error shows how much the average score varies between samples. A confidence interval is the range that probably contains the true score.
- When two variants answer the same questions, use a paired-difference test, which compares the two scores question by question. Question difficulty is correlated across variants, and pairing uses that correlation to reduce variance at no cost.
- Resample non-deterministic outputs and average per question.
- Run a power analysis first. It calculates the sample size that can detect the effect you care about.
Judging by eye is not evidence, even if the company's new prompt "looks clearly better on these twenty cases".
Staged Exposure for Risky Swaps
Standard deployment practice exposes risky changes in stages. Model-version migrations need this most, because behaviour can change broadly and silently. In a shadow run, the new variant handles real traffic, and its outputs are compared offline. Users do not see these outputs. A shadow run tests behaviour on real traffic with zero risk to users. In a canary, a small slice of live traffic gets the variant first. Guard metrics are watched, and a rollback is ready. The company shadows a newer model on live disputes before any canary.
Shadow runs and canaries both need versioned prompts and configurations with clean rollback. An experiment that cannot be reversed cleanly is a deployment. After a migration is complete, set a new baseline for every downstream measurement. Comparisons against the old model's baseline are now meaningless.
In the exam
Questions show a team like the company's that is about to do something that looks efficient but is wrong.
- bundling two changes into one test
- shipping on a passing offline suite
- declaring a winner from a handful of cases
When a question offers a clear improvement on a small sample, the right answer names sample size and significance.
The Improvement Loop
The exam expects this full loop: measure, hypothesise, change one thing, eval offline, and A/B test online with success and guard metrics. Then ship or revert, and make the losing variant's failures into 4.2 regression cases. Teams skip the last step. The last step is what lets the loop build on earlier results instead of repeating them.
Exam Traps
Bundling a model migration with a prompt rewrite because the system is being touched anyway
Two variables moved at once means no attribution: a win cannot be credited, the cause of a loss cannot be found. Test the model change alone first, then test the prompt rewrite separately against the new baseline.
Treating a passing offline suite as permission to release a user-facing change to everyone
The eval set approximates production; the A/B against the live baseline is the proof of real-world impact. Skipping it treats the approximation as fact while every user is exposed to the change.
Declaring a winner because the difference looks clear on a handful of cases
LLM variance makes small samples noisy, and a difference that looks clear is often within noise. The right answer names the methods: paired-difference tests, standard errors, and a power analysis run before the experiment.
Offering a rollback plan as a substitute for staged exposure
Shipping everything at once with a rollback still exposes all users to an unvalidated change, and detection after the fact is the most expensive way to learn it failed. Shadow runs and canaries exist to collect the evidence before users are exposed.
Practice Scenario
A team wants to migrate a support agent to a newer model version and simultaneously rewrite the system prompt, since they are touching it anyway. They must be able to say which change caused any shift in behaviour, and no change may reach every user untested. What is the right approach?
Build Exercise
Write the Experiment Plan for Your Next Change
What you'll learn
- How to isolate a single experimental variable and hold the rest constant
- How to pair success metrics with guard metrics and their baselines
- How to justify a sample size with a power analysis
- How to choose an exposure strategy proportionate to the risk of the change
- How to define the decision rule, rollback, and regression-case creation in advance
- Name the single variable the experiment changes, and what is deliberately held constant
Why: Attribution requires isolation — the plan should make it impossible to add a second change later without anyone noticing.
You should see: One named variable and an explicit held-constant list covering prompt, model, retrieval, and parameters.
- State the success metric and every guard metric, with the current baseline for each
Why: A variant that improves the success metric while breaching a guard metric has failed — the experiment must be able to detect that outcome.
You should see: One success metric plus guards (cost per task, tail latency, safety rates), each with a measured baseline.
- Justify the sample size: what effect size matters, and what does a power analysis say you need to detect it
Why: If you decide the sample size after seeing results, random noise can be mistaken for a real effect.
You should see: A minimum effect size worth acting on, and the case count required to detect it with confidence.
- Choose the exposure strategy (offline only, shadow, canary, full A/B) and defend the choice against the risk of the change
Why: Exposure should be proportionate: a copy tweak and a model migration do not deserve the same blast radius.
You should see: A named strategy with one sentence tying it to the risk class of the change.
- Define the decision rule in advance (ship, revert, or iterate), the rollback mechanism, and where failures from the losing variant will become regression cases
Why: Decision rules written after the results arrive are rationalisations; and the regression-case step is what lets each experiment add to the results of earlier ones.
You should see: An if-then decision rule, a tested rollback path, and the destination population for the failures.
Sources
- A statistical approach to model evaluations — Anthropic
- Define success criteria and build evaluations — Anthropic
- Choosing the right model — Anthropic
- Claude Certified Architect Professional Exam Guide (v1.0) — Domain 4 — Anthropic