What You Need to Know
Diagnosis is a fixed method that you follow, not a search for a ready answer. You find why a Claude system fails before you change anything. In a prompt failure, the instructions are the problem. A hallucination is confident fabrication. The model invents entities, citations, figures or clauses, and they are fluent enough that a non-expert would accept them. In a model mismatch, the task exceeds the model tier.
This lesson uses one made-up example throughout: a hotel chain whose Claude assistant answers guests from retrieved passages of its room descriptions, room rates and booking rules. After the chain loads its winter rates, the assistant starts to describe a non-refundable rate as refundable. The model and the latency are unchanged.
The Diagnostic Order
- Trace and reproduce. A trace records the inputs to the request (prompt version, retrieved context, tool results) and the output. Domain 3 built this tracing system, and Domain 4 uses it. One bad output proves nothing in a non-deterministic system. So confirm a pattern across cases, not a single run. The chain collects the trace of every wrong refund answer since the rate load.
- Isolate the stage. Let the evidence point to a pipeline stage before you name a failure class. The chain's errors began with the rate load, and the model and latency did not change. This evidence points to retrieval and indexing before anyone says the word "hallucination".
- Classify against the signatures (the typical symptoms of each class) below: prompt failure, hallucination (grounding or generation), model mismatch. Each has a recognisable signature.
- Run the cheapest decisive test for the hypothesis: the check that confirms or excludes the hypothesis.
- Fix at the failing layer, never at the layer that is merely easiest to change. The reflex fixes, "upgrade the model" and "rewrite the prompt harder", are conclusions without a diagnosis. They skip steps 2 to 4, so they can change a layer that never failed. This objective's wrong options are built from these reflex fixes.
- Create regression cases (objective 4.2) from the failing inputs, so the diagnosis remains after the incident ends.
Prompt Failure
The signature: format drift, violated output contracts (outputs that break the specified format), rules ignored or applied inconsistently, and behaviour varying with phrasing on identical inputs. Failures cluster where the instructions do not fully specify the behaviour. The chain's booking replies switch between list and paragraph as the wording changes.
The decisive test: a clarified, structured prompt (explicit rules, delimited data, an exact-format example) fixes the failing cases while everything else stays constant. Anthropic's consistency guidance lists these methods.
- Specify the output format precisely.
- Constrain with examples.
- Use structured outputs where schema conformance must be guaranteed.
Fix the prompt at the cheapest layer of Domain 2's ladder (its ordered list of prompt techniques): clarity, then examples, then reasoning. A model upgrade is the wrong fix. A larger model also follows ambiguous instructions, at a higher cost.
Hallucination: Grounding or Generation
In a grounding failure, the context was wrong, stale or missing, and the model answered faithfully from it. The cue: wrong answers correlate with events in the corpus, the document collection that retrieval searches (a refresh, a migration, the chain's rate load), or with degraded retrieval-quality signals. The fix is upstream in the Domain 3 pipeline: re-index (rebuild the search index), re-embed (compute the document embeddings again), repair retrieval. No instruction fixes the context.
In a generation failure, the trace shows the right passages, and the answer contradicts or exceeds them. On an older rate, the retrieved passage says non-refundable, yet the assistant says that the rate is refundable. The fix is a set of grounding techniques. Anthropic's reduce-hallucinations guidance lists these methods.
- Explicitly allow the answer "I don't know". This option drastically reduces false information.
- Extract word-for-word quotes first for long documents.
- Require citations, and retract any claim without a supporting quote.
- Restrict answers to the provided text.
- Use chain-of-thought verification (Claude explains its reasoning step by step before the final answer) or best-of-N verification (run the same prompt several times and compare the outputs).
Then measure faithfulness, so the rate is a number, not an anecdote.
Grounding failures and generation failures are fixed in different places. One hallucination fix for every wrong answer guarantees that half the fixes go to the wrong layer. This one-fix approach is the classic wrong answer.
Model Mismatch
The signature: uniformly mediocre quality on the hard cases across prompt variants. Failures persist with a clean prompt and verified-correct context. Quality changes with task difficulty, not with phrasing. The chain's assistant might fail only on group bookings that mix several room types and rates, whatever the prompt is.
The decisive test: run the failing set on a stronger tier. If the stronger tier meets the quality bar, the gap is capability. The upgrade then becomes a diagnosed decision with an eval that the upgrade must pass. Domain 2's right-sizing and Anthropic's upgrade guidance give the steps.
- Benchmark with your actual prompts and data.
- Compare accuracy and edge-case handling.
- Weigh cost.
If it does not, the problem was never the model. Resume the search upstream.
Mismatch happens in both directions. Consider a premium tier that is far above a simple task, like the chain's questions about check-in times. This case has the same diagnosis and the opposite fix. Objective 4.5 covers downsizing.
Reading the Cues
| Cue in the question | Likely class | Cheapest decisive test | Failing layer |
|---|---|---|---|
| Varies with phrasing on the same inputs; format drift | Prompt failure | Clarified, structured prompt, all else constant | Prompt |
| Wrongness correlates with a corpus refresh or re-import | Hallucination (grounding) | Inspect retrieved passages in the trace for the failing cases | Retrieval/indexing pipeline |
| Right passages retrieved, answer contradicts or exceeds them | Hallucination (generation) | Grounding discipline on the failing set; measure faithfulness | Prompt-level grounding |
| Fails the hard cases under every prompt variant, context verified | Model mismatch | Run the failing set on a stronger tier | Model tier |
Inconsistency means divergent outputs for the same input class. Inconsistency is usually a symptom of one of the failure classes, not a class of its own.
- Prompt failure: instructions that do not fully specify a boundary.
- Grounding: retrieval returning different context per run.
- Mismatch: genuine variance at the edge of capability.
The method is unchanged: trace, cluster the divergent cases, and find what differs between runs. Self-contradiction within one response is a reliability warning sign. It is worth its own regression cases.
Here, bias appears as a detection task: disaggregate (split) eval results across user groups and content types. A system that is accurate on average but systematically wrong for one population is failing. The report should say so. Detection is Domain 4's job. The governance response belongs to Domain 5.
In the exam
This objective punishes fluent guessing hardest. The exam guide's Domain 4 sample sits here: a RAG system returns confident but incorrect answers after a document refresh, latency and model version unchanged. RAG means retrieval-augmented generation: the model answers from retrieved passages. The right answer is the retrieval/indexing step, as at the hotel chain. The sample is a small example of the diagnostic order: evidence before conclusions, stage before failure class, cheapest decisive test before any fix.
What the Sample Question Asks For
In the sample, the right answer is the most likely first place to investigate, not the best eventual fix. Its wrong options (silently changed model weights, temperature too low, a shrunken context window) share one flaw: nothing in the question points to them. When a diagnostic question gives you a correlation, the right answer follows it.
Exam Traps
Upgrading the model tier as the first response to wrong answers
It costs the most and leaves the symptom unchanged: a stronger model obeys ambiguous instructions more expensively and fabricates just as fluently from a broken index. The upgrade is only correct after the mismatch signature is confirmed by the stronger-tier test.
Rewriting the prompt to suppress hallucinations that began after a corpus refresh
That is patching downstream of the fault. The correlation implicates grounding: the model is answering faithfully from bad context, and no instruction restores passages the retrieval step no longer supplies.
Using temperature zero to stop incorrect answers
Temperature 0 is not fully deterministic, and none of the failure classes is diagnosed or fixed by it. Isolate the failing stage. Current state (checked 29 October 2026): Claude Opus 4.7 and later models reject a non-default temperature, and API version 2026-10-21 rejects any value.
Calling every incorrect output a hallucination and treating hallucination as one problem
The grounding/generation split decides where the fix goes: pipeline versus prompt-level discipline. Treating the classes as one guarantees fixing the wrong layer half the time, and the exam writes options for both layers to catch it.
Practice Scenario
A policy assistant starts inventing clause numbers two days after the policy corpus was re-imported. Traces on the failing cases show the retrieved passages no longer contain the cited sections; latency, model, and prompt are unchanged. What should the team do?
Build Exercise
Write the Diagnosis Runbook for One System
What you'll learn
- How to describe failure signatures in the language a trace shows
- How to classify failures against the four classes using correlation cues
- How to design the cheapest decisive test for each hypothesis
- How to route each confirmed diagnosis to its failing layer and owner
- How to practise the diagnostic order on a real incident and find the skipped step
- For a Claude system you know, list its three most likely failure signatures in production, in the language a trace would show — not in failure-class jargon
Why: Diagnosis starts from evidence, and the runbook must describe what the on-call engineer will actually see.
You should see: Three concrete signatures, for example: answers cite document sections the retrieved passages do not contain.
- Classify each signature against the classes: prompt failure, hallucination — grounding, hallucination — generation, model mismatch — and note the correlation cue that would alert you
Why: The class decides the failing layer, and the cue is how you reach the class from evidence rather than reflex.
You should see: Each signature tagged with a class and a cue: after a corpus event points to grounding, varies with phrasing points to the prompt, fails hard cases everywhere points to the tier.
- Write the cheapest decisive test for each: exactly what you would change or run, with everything else held constant, and what result confirms the diagnosis
Why: The decisive test is what separates diagnosis from guessing, and cheapest first is the correct order.
You should see: One test per signature with a confirm condition, for example: run the failing set on a stronger tier; if it meets the quality bar, the gap is capability.
- Name the failing layer each confirmed diagnosis points to, and who owns the fix (prompt, pipeline, tier choice)
Why: Fix-at-the-failing-layer only works if the runbook routes each diagnosis to an owner before the incident happens.
You should see: A routing table: class to layer to owner, with no diagnosis routed to whoever happens to be on call.
- Practise the order: for the last real incident you saw, follow trace, stage, class, test, fix, regression cases — and note where the team actually skipped a step
Why: Every team skips a step under pressure, and knowing which one your team skips is the most valuable line in the runbook.
You should see: A six-step walkthrough of a real incident with the skipped step named honestly.
Sources
- Reduce hallucinations — Anthropic
- Increase output consistency — Anthropic
- Choosing the right model — Anthropic
- Migrating to API version 2026-10-21 — Anthropic
- Claude Certified Architect Professional Exam Guide (v1.0) — Domain 4 — Anthropic