Validation, Retry, and Feedback Loops

What You Need to Know

An extraction pipeline asks Claude to turn a document into structured fields. Validation checks those fields against rules. A retry asks the model again. Production extraction systems fail. Documents arrive in unexpected formats, numbers do not sum correctly, and values appear in the wrong fields. A validation-retry loop lets the workflow correct those failures itself.

This lesson uses one made-up example throughout: an invoice extraction pipeline at an airline, the one the code samples show. The finance team uses it to read supplier invoices. Each extraction holds line items, a stated total and a department. On the first invoice, the line items sum to £450 but the stated total is £500. The second invoice names no department at all.

Schema Syntax Errors vs Semantic Validation Errors

Schema syntax errors are malformed JSON, missing required fields and wrong data types. tool_use with JSON schemas eliminates them when you set strict: true (Task Statement 4.3). Semantic validation errors have correct structure but incorrect values: line items that do not sum, dates in the wrong order, values in wrong fields. A schema cannot say that two numbers must agree. So these errors need validation logic outside the schema, and retry loops focus on them. The first invoice has a semantic validation error.

Retry-with-Error-Feedback

The correct retry pattern sends three pieces of information back to the model.

1. The original document, so the model has the source to re-examine
2. The failed extraction, so the model can see what it produced
3. The specific validation error, so the model knows exactly what went wrong

// Retry with error feedback
const retryMessages = [
  {
    role: "user",
    content: `Original document:\n${originalDocument}\n\n` +
      `Your extraction:\n${JSON.stringify(failedExtraction)}\n\n` +
      `Validation error: Line items sum to £450 but stated_total is £500. ` +
      `Please re-extract, ensuring all line items are captured.`
  }
];

This works far better than a naive retry, which sends no specific error. Without the specific error, the model has no guidance and usually repeats the mistake. With the error, the model can re-examine the document for missed line items, check field placement or recalculate the total. When the retry says the first invoice's line items sum to £450 and not £500, the model looks for the missed line item.

The Retry Effectiveness Boundary

A retry only lets the model look again at what the pipeline gave it. Retries are effective for four kinds of error.

• Format mismatches (wrong date format, inconsistent currency notation)
• Structural output errors (values in wrong fields, incorrect nesting)
• Misplaced values (data that exists in the document but that the model extracted into the wrong field)
• Mathematical errors (the model missed a line item that affects the total)

Retries are not effective in three cases.

• Information genuinely absent from the source document
• Data that exists only in an external document not provided to the model
• Fields requiring knowledge the model does not have

If a value is absent from the source, no number of retries will produce a correct value. Flag the extraction for human review, or return null if the schema allows null. The first invoice is fixable: the missed line item is on the page. The second invoice goes to a person.

Self-Correction Flow Design

You can also build self-correction into the schema itself.

calculated_total vs stated_total: extract both the model's sum of the line items and the document's stated total. When the two totals differ, you have a discrepancy flag without external logic. The first invoice would arrive with this flag.

{
  "line_items": [
    { "description": "Cabin cleaning", "amount": 150.00 },
    { "description": "Catering supplies", "amount": 300.00 }
  ],
  "calculated_total": 450.00,
  "stated_total": 500.00,
  "total_discrepancy": true
}

conflict_detected booleans: add booleans that flag contradictory information in the source document. One section might say "payment due: 30 days" and another section might say "payment terms: net 60". The model should extract both values and set conflict_detected: true rather than silently pick one.

detected_pattern Fields

The software team at the airline also runs a code review pipeline. For code review and analysis pipelines, add detected_pattern fields to structured findings. Each detected_pattern field records which specific code construct triggered the finding.

{
  "finding": "Potential SQL injection vulnerability",
  "severity": "critical",
  "detected_pattern": "string concatenation in SQL query",
  "file": "user_service.py",
  "line": 42
}

You can then analyse dismissed findings by detected_pattern. If developers consistently dismiss a pattern, such as "variable shadowing in nested scope", you likely need to refine the prompt for that pattern. This analysis gives a systematic improvement loop: extract, validate, collect dismissal data, refine prompts, repeat.

Pydantic as the Validation Layer

In a Python pipeline, Pydantic turns "validation failed" into the specific, per-field error messages that the retry loop needs. A Pydantic model does two jobs. Parsing enforces structure: types, required fields, enums. Validators enforce semantics. Here, semantics means the rules that a JSON schema cannot express, such as cross-field arithmetic or date ordering. Both kinds of failure appear in one ValidationError. Its machine-readable errors name the field and the broken rule. The validator below is the check that the first invoice fails.

import json
from pydantic import BaseModel, ValidationError, model_validator

class LineItem(BaseModel):
    description: str
    amount: float

class Invoice(BaseModel):
    line_items: list[LineItem]
    stated_total: float

    @model_validator(mode="after")
    def totals_must_match(self):
        calculated = round(sum(i.amount for i in self.line_items), 2)
        if abs(calculated - self.stated_total) > 0.01:
            raise ValueError(
                f"line items sum to {calculated} but stated_total is {self.stated_total}"
            )
        return self

try:
    invoice = Invoice.model_validate(tool_input)  # the tool_use input from the response
except ValidationError as e:
    errors = "\n".join(
        f"{'.'.join(map(str, err['loc'])) or 'invoice'}: {err['msg']}" for err in e.errors()
    )
    retry_message = (
        f"Original document:\n{original_document}\n\n"
        f"Your extraction:\n{json.dumps(tool_input)}\n\n"
        f"Validation errors:\n{errors}\n\n"
        f"Please re-extract, fixing the identified errors."
    )

The except branch is retry-with-error-feedback. Pydantic supplies the specific error, ready to format into the prompt. The limit on what a retry can fix still applies. If a validator fails on information that is absent from the source, the extraction needs human review, not a retry.

In the exam

The retry effectiveness boundary is the concept that the exam tests most heavily in this task statement. The exam presents both kinds of failure and expects you to identify which kind is fixable. The exam also tests that tool_use solves schema syntax errors but not semantic ones. The overlap with Task Statement 4.3 is intentional. The exam guide names Pydantic alongside JSON Schema in its hands-on exercise for this task statement. It reads: "when Pydantic or JSON schema validation fails, send a follow-up request including the document, the failed extraction, and the specific validation error".

Exam Traps

Practice Scenario

Build Exercise

Sources

• Claude Certified Architect Foundations Exam Guide — Task Statement 4.4 — Anthropic
• Tool Use (Function Calling) — Anthropic
• Structured Outputs — Anthropic
• Pydantic documentation — Pydantic
• Tool Use with the Messages API: Strict tool use (Claude Academy) — Anthropic
• CCAR-F Exam Guide v1.0: Task Statement 4.4 — Anthropic