RAG Pipeline Design: Chunking and Indexing

What You Need to Know

A RAG pipeline answers from a document collection in stages: ingest, chunk, embed, index, retrieve, optionally rerank, generate. A chunk is a passage cut from a document. To embed a chunk is to turn it into a list of numbers that represents its meaning. Embedding search compares chunks by meaning. Keyword search, such as BM25, matches exact tokens. To rerank is to put the retrieved chunks in a new order, most relevant first. Each stage has its own failure mode.

This lesson uses one made-up example throughout: a hospital group whose assistant answers staff from its safety procedures, supplier contracts and equipment catalogue. Retrieval quality limits generation quality. If the right passage never reaches the context, the best model can only guess. When an earlier stage fails to bring the right passage into the context, the failure appears as hallucination in the answer. Design works stage by stage, and so does diagnosis.

Chunking Is a Trade-off

A chunk is the unit that retrieval finds. A chunk that is too small loses needed context. For example, a table split from its caption, or a clause split from its heading, becomes unfindable or misleading. A chunk that is too large lowers relevance and spends tokens on text that is not needed. A whole-document chunk matches everything weakly. Structure-aware chunking respects sections, headings and tables. With a modest overlap to guard against losses at chunk boundaries, it performs better than fixed-size splitting when the document structure is reliable. Many real documents, such as plain text or PDFs without clear headings, do not have a reliable structure. For those documents, fixed-size splitting with overlap is often the default choice in production. The hospital group's supplier contracts have consistent headings, and the meaning is in the structure, so the group keeps each contract clause with its heading.

Contextual Retrieval

The hospital group's chunk "repeat the check after a power failure" is unfindable until it says which procedure it belongs to. Many retrieval failures have that root cause: context that is lost at the chunk boundary. Before embedding and indexing, contextual retrieval adds a short explanation to the start of each chunk. The explanation is specific to that chunk and places it in its document. The hospital group's chunk gets the name of its safety procedure. Anthropic's published benchmark shows that contextual retrieval measurably reduces retrieval failures. The gains grow as contextual BM25 and reranking are added. The rule to remember is this: restore each chunk's lost context before you index it.

Index for How It Will Be Asked

Embedding search often misses exact strings. An exact part number needs BM25's token match. A question such as "why does the pump alarm sound at night" needs semantic similarity. Corpora with identifiers, part numbers, error codes or clause references need hybrid indexing: embeddings plus keyword. Metadata filters use stored facts about each document (date, source, product) to narrow the candidate set before similarity ranks it. Pure embedding search over an identifier-heavy corpus returns passages that are thematically similar but wrong, and no chunk tuning fixes this. The hospital group's equipment catalogue, with its part numbers, needs both.

Freshness Is an Operational Property

A RAG pipeline is a data system and needs data-system operations. At the hospital group, every procedure update must reach the index.

• Refresh triggers re-embed and re-index, atomically. Atomically means as one step that completes fully or not at all. When documents are updated without a re-index, the index silently serves the old corpus. Answers cite outdated content with full confidence.
• Pin the embedding-model version everywhere. A corpus embedded with one model and queried with another degrades silently. No errors are thrown, and retrieval is wrong.
• Monitor retrieval quality. Recall is the share of the relevant passages that retrieval returns. Recall on a labelled query set (Domain 4 covers the measurement) turns index breakage into an alert (objective 3.4), not something that users discover.
• Evaluate retrieval separately from generation. A pipeline that can only be debugged end to end is expensive to debug. Separate measurement isolates a failure to one stage.

When RAG Is Not the Answer

Retrieval is not the default. A corpus that is small and stable enough for the cached prompt is faster and cheaper to serve whole, and it has no retrieval failure. Domain 2's decision about what belongs in the context comes before pipeline design. The first decision is whether the pipeline should exist. One short, stable document at the hospital group needs no pipeline.

In the exam

The exam tests both design and diagnosis. Its favourite failure pattern comes from the guide's own sample: confident wrong answers immediately after a document refresh. At the hospital group, the assistant quotes the old safety procedure after an update. The right answer investigates the retrieval/indexing step first, because it is the stage most likely to return irrelevant or stale chunks.

Dated Figures

The contextual retrieval benchmark figures and the current token threshold for cached prompts are dated. They are in the Domain 3 quick reference.

Exam Traps

Practice Scenario

Build Exercise

Sources

• Introducing Contextual Retrieval — Anthropic
• Effective context engineering for AI agents — Anthropic
• Building with the Claude API: Text chunking strategies — Anthropic
• Claude Certified Architect Professional Exam Guide (v1.0) — Domain 3 — Anthropic
• CCAR-P Exam Guide v1.0: Sample 3 answer key — Anthropic