Batch Processing Strategies

What You Need to Know

The Message Batches API takes many requests in one submission and returns the results later. It is a cost optimisation tool with hard constraints. A synchronous call sends one request and waits for its answer. In a blocking workflow, someone or something waits for the result. A latency-tolerant workflow uses its results later.

This lesson uses one made-up example throughout: the development team at a telecoms company. Its pre-merge check reviews each pull request while a developer waits. That work is blocking. An overnight job writes a technical debt report on the codebase, and the team reads it next morning. That work is latency-tolerant.

Message Batches API: The Facts

• 50% cost savings compared to synchronous API calls
• Up to 24-hour processing window: results may arrive in minutes or take up to 24 hours
• No guaranteed latency SLA: an SLA (service level agreement) is a promise to deliver within a stated time. You cannot rely on results arriving within any specific timeframe
• No multi-turn tool calling within a single batch request: the model cannot execute tools mid-request and use the results to continue processing
• custom_id fields for correlating request/response pairs: each request in a batch gets a unique identifier used to match it with its response

The Matching Rule

Use the synchronous API for blocking workflows and the batch API for latency-tolerant ones. A batch has no latency guarantee and can take 24 hours. So anyone who waits for a batch could wait up to 24 hours. An overnight job loses nothing when it waits, and it costs half as much.

Blocking workflows include:

• Pre-merge checks in CI/CD
• Real-time code review feedback
• Any workflow where developers are blocked pending completion

Latency-tolerant workflows include:

• Overnight technical debt reports
• Weekly code audit summaries
• Nightly test generation runs
• Batch document extraction

The telecoms team keeps its pre-merge check synchronous, because a developer is waiting. It batches the technical debt report.

The first call below is synchronous. The second call is a batch.

// Synchronous — developer is waiting for this
const preMergeReview = await client.messages.create({
  model: "claude-sonnet-5-5",
  max_tokens: 4096,
  messages: [{ role: "user", content: prDiffContent }]
});

// Batch — results consumed tomorrow morning
const batchRequest = await client.messages.batches.create({
  requests: technicalDebtDocuments.map((doc, i) => ({
    custom_id: `debt-report-${i}`,
    params: {
      model: "claude-sonnet-5-5",
      max_tokens: 4096,
      messages: [{ role: "user", content: doc }]
    }
  }))
});

Multi-Turn Tool Calling Limitation

The batch API does not support multi-turn tool calling within a single request. You cannot:

• Define tools and have the model call them mid-request
• Process tool results and continue the conversation within the same batch item
• Run agentic loops (the model calls a tool, reads the result and continues) within a single batch request

A workflow that needs to execute a tool during processing must use the synchronous API for that step. If the telecoms team's overnight job queried its issue tracker in the middle of a file, that step would run synchronously.

SLA Calculation

Plan batch schedules for the full 24 hours. The 24 hours is a maximum window, not a delivery guarantee. A batch that does not finish inside that window returns as expired. Plan the schedule for that worst case, and treat an expired batch as work to resubmit.

Suppose the telecoms team also batches the extraction of data from customer contracts. Contracts arrive all day, and the team promises each one a result within 30 hours. Work backwards:

• 30 hours total SLA minus the 24-hour worst case = 6 hours of buffer for collecting requests, validating inputs, or absorbing operational delays
• Submit batches every 4 hours within that buffer window, so that a new batch is always running
• Submitting every 6 hours leaves no margin at all

A document that arrives just after a submission waits for the next batch. If you submit every 6 hours, it can wait 6 hours and then take 24 hours to process. That uses all 30 hours.

Batch Failure Handling

Not all documents in a batch succeed. The correct way to handle failures has three steps.

1. Identify failures by custom_id. Each request has a unique identifier. Parse the batch results to find which custom_id values failed.

2. Resubmit only failures with modifications. If you resubmit the entire batch, you pay again for documents that already succeeded. Common modifications include:

• Chunking (splitting into smaller parts) oversized documents that exceeded context limits, such as the telecoms team's longest contracts
• Simplifying extraction prompts for documents with unusual structures
• Adding format-specific few-shot examples (input-output pairs included in the prompt) for documents that failed due to structural variety

3. Refine prompts on a sample set before batch processing. This early step maximises the number of documents that succeed on the first submission and reduces the cost of resubmissions.

The handler below retries only failed and expired results. The results arrive as JSONL, a format with one JSON object on each line.

// Poll until the batch has finished before reading results
let batch = await client.messages.batches.retrieve(batchId);
while (batch.processing_status !== "ended") {
  await new Promise(r => setTimeout(r, 60_000));
  batch = await client.messages.batches.retrieve(batchId);
}

// results() returns a JSONL async iterable, not an array — accumulate it.
// Treat `expired` as a failure too: that is what an overrun batch returns.
const failedIds: string[] = [];
for await (const result of client.messages.batches.results(batchId)) {
  if (result.result.type === "errored" || result.result.type === "expired") {
    failedIds.push(result.custom_id);
  }
}

// Resubmit only failures with modifications
const retryRequests = failedIds.map(id => {
  const originalDoc = documentsById[id];
  return {
    custom_id: `${id}-retry-1`,
    params: {
      model: "claude-sonnet-5-5",
      max_tokens: 8192,  // increased for oversized docs
      messages: [{
        role: "user",
        content: chunkIfNeeded(originalDoc)
      }]
    }
  };
});

Prompt Optimisation Before Batch Submission

The most cost-effective batch strategy is to refine prompts before submitting large volumes. The telecoms team's sample would cover each type of customer contract.

1. Sample set testing: Take 5-10 representative documents covering the range of formats, edge cases, and document types in your batch
2. Iterate on the sample: Refine your extraction prompts, add few-shot examples, adjust schema design until the sample set achieves high accuracy
3. Submit the full batch: With refined prompts, your first-pass success rate will be significantly higher
4. Handle failures: Resubmit only the failed documents with targeted modifications

Every failed document costs a retry, so this workflow cuts total cost. A 90% first-pass success rate on 1,000 documents means only 100 retries. A 60% rate means 400 retries. That is four times the resubmission cost.

In the exam

The matching rule is the single most tested concept in this task statement. In Question 11 of the sample questions, a manager proposes to switch everything to batch to get the saving. The correct answer keeps blocking workflows synchronous and batches only latency-tolerant ones.

The exam tests the tool limitation directly. A batch workflow that calls external tools during processing needs the synchronous API for that step. A scheduling question may work backwards from an SLA to a submission frequency.

Exam Traps

Practice Scenario

Build Exercise

Sources

• Claude Certified Architect Foundations Exam Guide — Task Statement 4.5 — Anthropic
• Message Batches API — Anthropic
• CCAR-F Exam Guide v1.0: Task Statement 4.5 — Anthropic