What You Need to Know
Every integration has two numbers: a latency requirement, such as a p95 response time, and a quality requirement. Accuracy controls protect the quality requirement. Reranking is one control: a second step that reorders the retrieved passages and keeps the most relevant ones. A verification pass is another: a second check of the answer before the system returns it. The skill is meeting both requirements at once.
This lesson uses one made-up example throughout: a logistics company whose website assistant answers customers' questions about their shipments. The assistant retrieves the customer's shipment records, reranks them, checks the current location with a tracking tool and answers. It has a p95 target and a threshold for answers about the wrong shipment. Any change must keep both.
The Budget Is a Sum
Latency is a sum: each stage on the request path adds its own time.
- Time-to-first-token (the wait before the first token of the response arrives) and generation length for each model call, including any extended thinking
- Each retrieval and reranking stage
- Each tool round-trip
- Each agent-loop iteration
A cut to the wrong term wastes the accuracy that the cut costs, because a small term barely changes the total. In a serial tool chain, the latencies add together, because each call waits for the call before it. Only genuinely independent calls can run in parallel. A step that uses the output of another step keeps its serial latency, however it is scheduled. The company's tracking check needs the shipment that retrieval found, so the tracking check stays serial.
Anthropic's latency guidance orders the options: choose the right model tier for the task, reduce input and output tokens, then stream for interactive interfaces. First find the dominant term. Traces (objective 3.4) measure how the total splits into terms. A fix aimed at any other term costs something and gives no visible improvement.
Accuracy Levers Pull the Other Way
Every accuracy control adds work, so each control has a latency cost.
- More retrieved context
- A reranking stage
- A verification pass
- More reasoning
- A bigger model
Every latency cut risks losing accuracy. Spend selectively: apply the expensive control where a wrong answer would be costly. Verifying high-value items and sending the rest through a faster path routinely works better than both verifying everything and removing verification everywhere. Verifying everything adds cost and latency to all traffic. Removing verification everywhere deletes a named quality requirement. The logistics company verifies answers about high-value freight shipments and sends a question about a small parcel through the faster path.
The Domain 2 mistake of moving to a smaller model without measuring accuracy reappears at the integration layer. A smaller, faster model is a legitimate latency fix only when accuracy is re-measured against the quality threshold afterwards. Until then, its error rate is unknown, so the change to a smaller model is a guess.
Streaming and Caching
Streaming returns tokens as the model generates them, so users see progress immediately and perceived latency improves. Total generation time is unchanged. Streaming suits interactive interfaces. A machine consumer that parses the complete response before acting waits for the last token, so streaming is irrelevant to it. The company's customer chat benefits from streaming. Its nightly job that labels shipment records does not.
Prefix caching is the rarer case that gives two benefits at once. A stable prefix (system prompt, tool definitions, reference corpus) makes repeat requests both cheaper and faster to first token. Put stable content in the cached prefix. Do not put volatile content there, because volatile content changes the prefix on every request. The logistics company puts its system prompt, tool definitions and delivery terms in the prefix. The retrieved shipment records and the customer's question come after the prefix.
The Justification Names Both Numbers
"Faster is better" and "more accurate is better" are not justifications. Domain 4's evals supply the accuracy threshold. The traces from objective 3.4 supply the split of latency into its terms. The configuration decision cites both. For the logistics company, the justification is: meets the stated p95 while holding the measured wrong-shipment rate.
In the exam
The wrong options sacrifice one named requirement to improve the other. For example, a wrong option drops verification to meet the SLA, or it verifies everywhere and misses the SLA. The right answer meets both, and its justification names both: "Meets the stated p95 while holding the measured wrong-shipment rate".
- Identify the stated latency requirement and the stated quality requirement.
- Eliminate every option that sacrifices a named requirement to improve the other.
- Among the remaining options, pick the one that addresses the dominant budget term.
When the consumer in the question is a pipeline, a parser or a batch job, the streaming option is necessarily wrong.
One-Sided Justifications Are the Wrong Answer
When a question names both an SLA and a quality threshold, every option with a justification on one side only is a wrong option. The right answer restructures the design: routing by value or risk, caching the stable prefix (the start of the prompt that does not change), parallelising independent calls. It never deletes a control that the use case names as critical.
Sourcing Notes
Domain 2 covers the caching mechanics. The published magnitude figures for caching are figures from a stated date, and the Domain 3 quick reference holds them.
Exam Traps
Dropping reranking or verification in a compliance- or safety-critical workflow to hit a latency target
A named requirement was sacrificed for the other. The right answer restructures the path with routing, caching and parallelism rather than deleting the accuracy control the use case names as critical.
Offering streaming as the latency fix for a machine consumer or batch pipeline
Streaming improves perceived latency for humans watching tokens arrive; total generation time is unchanged. A parser consuming the full response gains nothing, so its p95 does not move. Trim the generated tokens or choose a model of the right size instead.
Running the expensive verification pass on every request when only a subset is high-stakes
Verifying every request increases cost and latency across all traffic. Value- or risk-based routing applies the expensive path only where its price is justified.
Parallelising a chain where step B consumes the output of step A
Only independent calls parallelise. Dependent chains keep their serial latency however they are scheduled, and choosing the option shows that the candidate never split the budget into its terms.
Practice Scenario
A claims-triage integration must respond within a 2-second p95 and must not mis-route high-value claims (a measured accuracy threshold). The current path: retrieve policy context (600ms), rerank (400ms), large-model classification with a verification pass (1.8s). Which change best meets both requirements?
Build Exercise
Decompose a Latency Budget Against Its SLA
What you'll learn
- How to break a request path into its latency terms and find the dominant one
- How to name the requirement each accuracy control protects
- How to design routing, caching, and parallelism changes that meet both numbers
- How to write a configuration justification that cites the SLA and the quality threshold
- Take an integration with a latency SLA — or set one — and list every stage on the request path with its measured or estimated time
Why: Latency is a sum, and optimisation aimed anywhere but the dominant term wastes its cost.
You should see: A stage-by-stage table whose times add up to the observed end-to-end figure.
- Identify the dominant term, and mark which stages are serial and which could run in parallel without dependency violations
Why: Parallelism cuts latency at no cost only for genuinely independent calls; a dependent chain keeps its serial cost however it is scheduled.
You should see: One term circled as dominant, and each stage marked serial or parallelisable with the dependency named.
- Name every accuracy control on the path — retrieval depth, reranking, verification, model tier — and the requirement each one protects
Why: A control with no named requirement is a candidate for removal; a control with one must not be removed by cuts across all traffic.
You should see: A control-to-requirement mapping, with any unmapped control flagged.
- Design the cheapest change that meets the SLA without deleting a control tied to a named requirement — consider routing by value or risk, caching the stable prefix (the start of the prompt that does not change), and parallelising independent calls
Why: Restructuring is better than sacrificing a requirement: the right answer meets both numbers rather than trading one for the other.
You should see: A proposed change with its expected effect on the dominant term and no named requirement harmed.
- Write the justification sentence naming both numbers: the p95 achieved and the accuracy threshold met, with where each number comes from
Why: Configurations justified on one side of the trade-off are the wrong option in the exam, and an unjustifiable change in production.
You should see: One sentence citing the SLA, the quality threshold, traces as the latency source, and evals as the accuracy source.
Sources
- Reducing latency — Anthropic
- Prompt caching — Anthropic
- Introducing Contextual Retrieval — Anthropic
- Claude Certified Architect Professional Exam Guide (v1.0) — Domain 3 — Anthropic