Chunking and reranking for advanced RAG

What do chunking and reranking do in RAG?

Retrieval-augmented generation (RAG) pairs a language model with an index it can search at answer time. The original paper combined a pre-trained generator with "a dense vector index of Wikipedia" reached through a neural retriever.^[1] In a business setting the index holds your policies, manuals and contracts.

Two steps decide what the model reads:

• Chunking splits each document into the units that get embedded and stored. A unit that was never stored can never be retrieved.
• Reranking takes the candidates the first search returns and puts them in a better order, so the best passages reach the model first.

Get either wrong and the model answers from a fragment or from a passage that only looks relevant.

Why does fixed-size chunking fail?

Because it cuts where the counter says, not where the meaning ends. Common failures:

• A compliance clause is split from its condition, so a conditional duty reads as absolute.
• A numbered procedure is spread across several chunks, and only some of them are retrieved.
• A table loses its header row, so its numbers lose their meaning.

Fixed windows are still a fair first baseline. Test them against real questions before you build on them.

Which chunking strategies work better?

Split along the structure the document already has. LangChain's documentation describes three families: length-based splitting by tokens or characters, recursive splitting that tries to keep paragraphs intact and falls back to sentences, and document-structure splitting on Markdown headers, HTML tags, JSON elements or code blocks.^[2]

Two further patterns help in practice:

• Small-to-big retrieval. Index small, focused chunks for matching, but pass the larger parent section to the model, so the match is precise and the context complete.
• Contextual chunks. Add the document title, section heading or a short summary to each chunk before embedding it. Anthropic describes this as "prepending chunk-specific explanatory context to each chunk before embedding," and applies the same context to a keyword (BM25) index.^[3]

Chunking is a content-modeling decision. The aim is the smallest unit that still answers a likely question on its own. Clean parsing comes first; our guide to normalizing unstructured data for LLMs covers that step.

Why add a reranker after retrieval?

Because the fast first search trades precision for speed. Most RAG systems retrieve with a bi-encoder: an embedding model that encodes the question and each document separately, so documents can be indexed in advance.

A reranker reads the question and one candidate together and outputs a relevance score. The Sentence Transformers documentation explains that cross-encoders do better because "they perform attention across the query and the document," but that scoring thousands or millions of pairs this way "would be rather slow."^[4] Its suggested pattern is to retrieve a list of, for example, 100 hits and rerank only those.^[4]

The approach is well tested. Nogueira and Cho applied BERT to passage reranking and reported that their system topped the MS MARCO passage retrieval leaderboard, beating the previous best by 27% (relative) in MRR@10.^[5] Large language models can act as rerankers too: Sun and colleagues found that properly instructed models gave competitive results on retrieval benchmarks, and distilled that ranking ability into smaller models.^[6]

What can a reranker fix, and what not?

A reranker reads exact wording, so it can:

• Down-rank negations, such as a passage saying a rule "does not apply" to the group the question asks about.
• Prefer passages that answer over passages that only share the topic.
• Catch wording gaps that embeddings blur, when trained or tuned on similar data.

It cannot add a passage the first search missed. If recall is the problem, fix chunking, add a keyword signal (see our guide to hybrid search with dense and sparse embeddings) or widen the candidate set.

How do chunking and reranking interact?

They work as a chain:

1. Chunks too small: a definition is split across chunks and its qualification never enters the candidate set, so the reranker cannot help.
2. Chunks too large: each candidate carries mixed content, and the reranker scores noise along with the answer.
3. Chunks with context: titles and headings give the reranker more signal about where a passage sits.

Anthropic's tests are one example of the combined effect. Contextual embeddings plus contextual BM25 cut the top-20 retrieval failure rate by 49%, and adding reranking took the cut to 67%, measured as 1 minus recall@20 on their datasets.^[3] Your corpus will behave differently, so measure it.

How do you measure whether it helped?

Build a test set of real questions with known source passages. Then:

• Measure recall at your cut-off (the share of questions whose source passage appears in the top results) before and after each chunking change.
• Measure answer accuracy with and without the reranker.
• Track latency and cost per query alongside both.

Order also matters for the model itself. Liu and colleagues found performance "is often highest when relevant information occurs at the beginning or end of the input context."^[7] A reranker that puts the best passage first works with that pattern. To check answers against their sources afterwards, see AI hallucination verification pipelines.

Where should a team start?

1. Parse documents cleanly and chunk on their structure.
2. Add title and section context to each chunk.
3. Write a test set and measure retrieval before touching prompts.
4. Add a reranker over a candidate list sized by your test results.
5. Tune prompts and models only after retrieval stops losing passages.

The full data side is covered in our guide to enterprise RAG architecture. Our data infrastructure for AI service covers these steps as part of our AI consulting work.