Quick Reference: Domain 4 — Evaluation, Testing & Optimisation

Order of operations: define success criteria → design evaluations → build. Metrics chosen after launch match whatever the build happened to achieve, not the business need.

Every metric has three parts: definition + measurement method + target with baseline. "Accurate" is not a metric.

• Success versus guard metrics: what a change should improve versus what it must not damage (cost per task, tail latency, safety rates). Without guard metrics, regressions pass unchecked.
• Leading signals (drift indicators) warn early; lagging proof (formal eval results) is the evidence for decisions. You need both.
• Proxy trap: any proxy optimised hard enough stops tracking the outcome it proxied. Audit proxies against the real outcome; stop using them when they no longer move with the outcome.
• Anthropic's named dimensions include task fidelity, consistency, latency, and price, with criteria that are specific, measurable, achievable, and relevant.
• Latency vocabulary: baseline latency is the time to process the prompt and generate the response; time to first token is the time to the first token of that response, and it is the only one streaming moves.

Four populations, each covering different risks: representative (the bulk, mirroring the production distribution) · edge (irrelevant data, overly long inputs, poor user input, ambiguous cases) · adversarial (injection attempts, leading questions, malformed input) · regression (one created per production failure).

Golden-set discipline: versioned · held out of the prompt · refreshed from production traffic. Held out means out of the few-shot examples, not only out of the training data.

The pairing rule: deterministic where possible, judge where not, humans to calibrate the judge and audit the ambiguous cases.

• Anthropic's design principles: be task-specific, include edge cases, automate when possible, and prioritise volume — more questions on slightly less informative automated grading beat fewer hand-graded ones. Best practice is a different model as judge than the one that generated the output.
• Statistics, not round numbers (Anthropic's statistical evals research): report the standard error of the mean with confidence intervals; cluster standard errors on related questions, because clustered errors on popular evals can be over three times the naive ones; resample non-deterministic answers and average per question; run a power analysis before choosing the sample size.
• Both extremes appear as wrong options. An uncalibrated judge adds no evidence. Grading everything by hand uses human review where it is not needed.

Offline evals gate; online experiments validate. A change that fails the suite goes no further. Passing it is necessary but not sufficient for a user-facing behaviour change, because the eval set is an approximation of production, while real users are the actual production traffic.

• One variable at a time: prompt OR model OR retrieval. "Since we're touching it anyway" is how an experiment becomes a deployment.
• Every experiment carries success and guard metrics. A variant that improves the main metric but damages a guard metric has failed.
• Evidence bar: a paired-difference test when both variants answer the same questions, standard errors, resampled outputs, and a power analysis run first. Question-score correlation on frontier models typically runs 0.3 to 0.7, so pairing is free variance reduction. "Clearly better on twenty cases" is not evidence.
• Staged exposure, proportionate to risk: shadow (new variant runs on real traffic, outputs compared offline, zero user exposure) → canary (small share of live traffic, guard metrics watched, rollback ready) → full A/B. A rollback plan substitutes for neither.
• Model-version migrations shadow by default, and once a migration is complete, re-baseline every downstream measurement: the baseline changed, so old comparisons are meaningless.
• Anthropic's upgrade order: build benchmark tests for your use case → test with your actual prompts and data → compare accuracy, response quality, and edge-case handling → weigh performance against cost.
• The loop: measure → hypothesise → change one thing → eval offline → A/B online → ship or revert → the losing variant's failures become regression cases.

The objective that punishes fluent guessing hardest, and the guide's own Domain 4 sample sits here: confident wrong answers after a document refresh, model and latency unchanged, where the right answer is the most likely first place to investigate (retrieval/indexing) rather than the best eventual fix.

Diagnostic order: trace and reproduce → isolate the stage → classify against the signatures (the typical symptoms of each class) → cheapest decisive test → fix at the failing layer → create regression cases.

• Generation-failure repertoire (Anthropic's reduce-hallucinations guidance): allow "I don't know" explicitly, extract word-for-word quotes first on long documents, require citations and retract any claim without a supporting quote, restrict answers to the provided text, verify with chain-of-thought or best-of-N.
• Prompt-failure repertoire (increase-consistency guidance): specify the output format precisely, constrain with examples, use structured outputs where schema conformance must be guaranteed.
• Mismatch happens in both directions. A premium tier that is far more capable than a simple task needs is the same diagnosis, with the opposite fix. Lesson 4.5 covers moving to a smaller model.
• Inconsistency is a symptom, not a class: instructions that do not specify enough, retrieval varying per run, or genuine variance at the edge of capability. Trace, cluster the divergent cases, find what differs between runs.
• Bias detection is disaggregation: split eval results across user groups and content types. A system accurate on average and wrong for one population is failing. Detection belongs to Domain 4; the governance response belongs to Domain 5.
• Common wrong options: the model upgrade made without a diagnosis, prompt-patching a broken index, temperature zero as a correctness fix, and one word ("hallucination") for every wrong answer.

Two strict rules apply before and after every optimisation. Measure first: traces plus usage data name the dominant term (the largest contributor to cost or latency). Re-gate with evals: "cheaper and still passes the bar" is the only win condition.

Figures as of 30 September 2026. Re-verify before quoting; the reasoning outlives the numbers:

• Batch processing: 50% off all usage (input, output, and special tokens). Most batches finish in under 1 hour. Results are readable when every request completes or after 24 hours, whichever comes first, and a batch expires at 24 hours. Results stay downloadable for 29 days after creation. One batch is limited to 100,000 requests or 256 MB, whichever is reached first. stream is unsupported inside a batch. Caching discounts stack with the batch discount, with best-effort hit rates typically between 30% and 98%.
• Prompt caching: reads cost 0.1× base input price on most models (0.05× on Opus 5.5 and Sonnet 5.5, 0.025× on Fable 5.2, Mythos 5.2, Fable 5.1 and Mythos 5.1). Five-minute writes cost 1.25×, and one-hour writes cost 2×. The maximum is 4 breakpoints. The TTL is five minutes by default, and each use refreshes it at no cost. Minimum cacheable prefix (checked 14 November 2026): 512 tokens (Fable 5.2, Mythos 5.2, Fable 5.1, Mythos 5.1, Opus 5.5, Sonnet 5.5, Haiku 5.5, Opus 5, Fable 5, Mythos 5), 1,024 (Sonnet 5, Sonnet 4.6, Opus 4.8), 2,048 (Opus 4.7), 4,096 (Haiku 4.5, Opus 4.6, Opus 4.5). Below the minimum the request runs uncached, with no error.
• Effort levels, highest token spend first: max · xhigh · high · medium · low. high is the default on every model that supports effort except Opus 5.5 and Haiku 5.5, which default to medium. Sending the default is identical to omitting the parameter. Effort shapes all tokens: text, tool calls, and thinking. Anthropic states that tuning effort is often a better lever than switching models. Hold it constant inside a cached conversation, because changing it can invalidate all or part of the cache.
• Prices per million tokens, input/output: Fable 5.2 $8/$40 · Opus 5.5 $4/$20 · Sonnet 5.5 $2/$10 · Haiku 5.5 $0.10/$0.50 (prompts up to 100,000 tokens; $0.50/$2.50 above). Halve each for batch. Fast mode provides up to 2.5× output speed at premium pricing (checked 13 November 2026). It is available on Fable 5.2, Mythos 5.2 and Opus 5.5, and in research preview on Opus 5.
• Errors worth recognising: 400 invalid_request_error (an over-long prompt returns this error) · 413 request_too_large (Messages 128 MB, Batch 256 MB, Files 500 MB) · 429 rate_limit_error · 500 api_error · 504 timeout_error · 529 overloaded_error. The official SDKs retry transient failures twice by default with exponential backoff, honouring retry-after.

Wrong options: batch offered for an interactive path, moving to a smaller model with no re-gate, and caching offered for window overflow, wrong answers, or a workload of unique one-off documents.

Domain 3 built the infrastructure — traces, sampling, aggregate metrics. Domain 4 runs the quality practice on top of it. Uptime does not measure wrongness: a system can be fully up, fast on average, and confidently incorrect.

• Continuous evaluation: sample production traffic and score it automatically — programmatic checks for deterministic properties, the calibrated judge for the rest. Quality becomes a trend line against the launch baseline instead of a single result on release day.
• Leading indicators, each mapped to a stage: token spend per request (inputs grew, or a prompt edit landed) · refusal rate (traffic shifted, or a policy prompt interacts badly with new inputs) · tool-failure rate (an upstream API changed) · input-length distribution (a new client integrated) · retrieval-quality signals (a corpus refreshed — the guide's own sample scenario, caught early).
• Alerts fire on guard metrics at tail percentiles, with thresholds derived from measured baselines. Alerts on averages fire late or never; alerting on everything trains the team to ignore alerts, so choose deliberately what not to alert on.
• Incident loop: alert → traces → diagnose with 4.4's order → fix at the failing layer → create regression cases → re-baseline if the baseline legitimately moved. If the team skips the regression cases, it works on each bug twice.
• Instruments as of mid-2026: the Usage and Cost Admin API reports usage grouped and filtered by model, workspace, API key, service tier, context window, inference_geo (data residency: global, us, or not_available), and speed. It reports in one-minute, one-hour, or one-day buckets (cost is daily only, in USD). Data is typically visible within five minutes. Its named use cases include monitoring product performance, cache efficiency, and budget alerting. The speed group-by and the speeds[] filter separate fast mode usage from standard usage (Anthropic, Usage and Cost API). The Usage and Cost API is now a legacy API that remains supported. New integrations should use the Usage and Cost Analytics API. Guardrails guidance adds continuous monitoring of outputs for signs of successful injection.
• Wrong options the exam deliberately includes: re-running the launch suite as proof of present health, waiting for concrete user complaints, and the undiagnosed model upgrade.