What You Need to Know
An operational incident is an event in which a production agent behaves in a way that nobody expected. To resolve an incident is a team skill. That skill covers who can see the evidence, what the first move is, and what the team keeps after the incident. Containment asks whether harm is ongoing. Diagnosis asks what is wrong. Blast radius is how far the failure reaches outside the team.
This lesson uses one made-up example throughout: a software company whose agent reads customer bug reports and assigns each one to an engineering team. Since last week, some reports arrive at the wrong team. Each team reviews its new reports before it starts work, so the errors stay internal and reversible. That week the company also loaded a new list of product components and updated its issue-tracker connector.
Read the Evidence First
When a production agent begins to behave unexpectedly, the first move is to read what happened. That means: read the inputs, the retrieved context, the tool calls and what they returned, the outputs, and the versions in use.
The tempting first move is to edit the agent's instructions. That move is wrong for three reasons, and the reasons add together. Nothing yet implicates the instructions. The edit changes the system under investigation. Also, the failing runs that would have identified the cause become unreproducible, so the company could no longer replay last week's reports. A fix applied before the failure is understood usually changes something and occasionally fixes something. Such a fix always destroys the evidence.
Containment Is a Different Decision From Diagnosis
Different facts decide containment and diagnosis. Blast radius decides containment. When an external effect is still accumulating, contain it immediately: money moving, messages sent, records altered outside the team's control. Revert to the last known-good version (the most recent version that worked correctly), disable the capability, or make the action wait for an approval. Then diagnose in calm conditions.
When a drop in quality is internal and reversible, it is more useful to keep it as evidence than to restore the system. The company's wrongly assigned reports are this kind of drop. If the agent sent wrong answers to customers, the team would contain first.
Containment is not a diagnosis either. A reverted system no longer shows the symptom, but nobody understands the cause. Containment that reverts everything at once also loses attribution, which is the knowledge of which change caused the failure. When several changes are reverted together, nobody learns which one was responsible.
Blast Radius Decides, Not Discomfort
Ongoing external effect: contain first, diagnose second. Internal and reversible: the failing cases are the evidence, and a rollback destroys them.
Change History Narrows the Search Before Any Trace Opens
A change causes most operational incidents. So the fastest question is: what changed, and when did the symptom start? Instruction versions, model versions, corpus refreshes, configuration changes, and dependency and connector updates all need a version and a timestamp. When you correlate the start of the symptom against that timeline, you eliminate most of the possible causes without changing the system.
When two changes arrive in the same time window, test them against the evidence one at a time, instead of reverting both. When two variables change together, nobody can say which one caused the symptom. So the company tests the component list and the issue-tracker connector separately. This is Domain 4's one-variable rule. Anthropic's configuration debugging uses the same method: remove variables until the behaviour changes. Then reintroduce them one at a time.
The Agent Investigates; It Does Not Adjudicate
AI-assisted tooling shortens root-cause work in five ways:
- reading large volumes of logs and traces
- reconstructing a timeline
- correlating a symptom against a deployment
- reproducing a failing case
- proposing hypotheses faster than a human lists them
Correlated telemetry makes this possible. It collects everything that a single prompt triggered into one connected record.
The tooling does not decide the answer. When you ask an agent why it behaved unexpectedly, it gives a coherent account of its own reasoning. That account is a hypothesis. It comes from the same context that produced the behaviour. It has no privileged access to the cause. It persuades because it is fluent. If the company's agent blames the new component list, that claim stays unproven until a human confirms it against the trace. Non-determinism makes confirmation harder. One clean re-run does not disprove a hypothesis, and one failing run does not prove it. Confirmation means a pattern across collected cases.
Enablement Is Built Before the Incident
Everything above assumes a capability that somebody designed in advance:
- traces that exist and are readable
- a pager-holder (the engineer on call) who can open the traces at 2am without waiting for an approval
- rollback that is available to more than one named individual
- a runbook that names the first three moves
- shared investigation tooling, rather than whatever someone assembles privately
A pager-holder without access to the traces has no first move, and no diagnostic discipline compensates for that. If only one senior engineer can open the traces, a 2am failure waits until morning.
What the Incident Must Leave Behind
A fix at the failing layer, verified against the real case that failed. A regression case in the suite that Domain 4 owns. A runbook update while the gap is recent. Any access gap closed or recorded.
In the exam
Domain 4 diagnoses a product's quality failures against evals and traces. Where Domain 4 meets this objective, the question asks about the practice and the people, not the classification. Reading the evidence first is most reliably the right answer in the domain. Editing the agent's instructions is most reliably the tempting wrong answer. The exam tests containment through blast radius, so read the question to learn what the failure is doing outside the team. The option that ships the fix and closes the ticket is the trap for candidates who nearly pass: it stops one step before the end.
Exam Traps
Rewriting the prompt or the shared instructions as the first move
Nothing yet implicates the wording, and the edit changes the system under investigation, so the failing runs become unreproducible. Correlate symptom onset against the change history and read what the failing runs did (tool calls, outputs, versions) before changing anything.
Reverting several recent changes at once as a precaution, with no ongoing harm to stop
Containment is warranted by blast radius, not by discomfort. Reverting everything at once may quieten the symptom while destroying attribution, so nobody can say which change caused the failure and the incident returns the next time either change is deployed.
Accepting the agent's account of its own behaviour as the root cause, or clearing a hypothesis on one clean re-run
The account comes from the same context that produced the behaviour and is persuasive because it is fluent, so it is a hypothesis to confirm against the trace. Under non-determinism a single run proves nothing either way.
Closing the incident with a fix and nothing else, or planning an investigation that assumes access nobody has granted
A fix with no regression case is not protected against the failure returning, and a team with no runbook update is no faster next time. An on-call engineer who cannot read traces or roll back has no first move, whatever the plan says.
Practice Scenario
An internal reporting agent has started skipping its source-verification step on a minority of nightly runs. The effect is internal and every affected report has been regenerated without loss. Two things were deployed this week: a model version bump and a rewrite of the shared project instructions. Nobody has opened a trace, and asked about a failed run the agent gives a fluent account of why it judged the step unnecessary. The team wants to know today which of this week's two updates caused the skip, before altering anything. What should the team do first?
Build Exercise
Write the Incident Page for One System
What you'll learn
- How to name the first three pieces of evidence an on-call engineer reads
- How to test blast radius to decide between containment and diagnosis
- How to make change history usable as a diagnostic instrument
- How to separate an agent hypothesis from a confirmed finding
- How to close an incident so the team is faster the next time
- For a production Claude system you know, name the first three things an on-call engineer reads when it misbehaves, and the access each one requires
Why: Every diagnostic move assumes evidence that somebody made readable in advance.
You should see: Three named sources in reading order, each with the permission or tool needed to open it.
- Record who holds that access today and how they obtain it at 2am, then flag every step that depends on one particular person being awake
Why: An investigation plan that assumes access nobody granted cannot be carried out.
You should see: An access list with at least one uncomfortable single point of dependency flagged.
- Write the containment decision rule: which failures are contained before diagnosis, which are diagnosed on live evidence, and the blast-radius test that separates them
Why: Containment and diagnosis answer different questions, and the exam tests whether you can tell which one the facts call for.
You should see: One rule a tired engineer can apply, with a worked example for each of the two cases.
- List every change stream that could move behaviour - instruction versions, model version, corpus refresh, configuration, dependency and connector updates - and say where each version and deployment time is recorded today
Why: What changed and when the symptom started narrows the search before a single trace is opened.
You should see: A table with one row per kind of change, with gaps shown wherever nothing is timestamped.
- Audit the last real incident the team handled: what regression case it left, what runbook update it left, and what access gap it exposed
Why: When an incident is closed with no artefact, its full cost is paid a second time.
You should see: Three answers, some of them blank, and a short list of what to write down before the next one.
Sources
- Debug your configuration — Anthropic
- Monitoring — Anthropic
- Best practices for Claude Code — Anthropic
- Claude Certified Architect Professional Exam Guide (v1.0) — Domain 7 — Anthropic