Picking a chunking strategy for documents an event team actually keeps

Two documents from the same folder. One is a 22 page post-show report with nine numbered sections, a contents page and consistent headings. The other is a signed exhibitor contract where clause 1 defines "Contracted Space" and clause 14 imposes a penalty for reducing it, with thirteen clauses of unrelated text in between. A chunking strategy for documents that treats those two the same way will work acceptably on the first and fail on the second, and the failure will be invisible because the system will still answer.

Chunking is the least glamorous decision in a retrieval build and it caps everything downstream. If the passage containing the answer never exists as a unit, no ranking model and no prompt recovers it.

What a fixed size chunker actually breaks

Start with the arithmetic, because it makes the behaviour concrete.

The 22 page report at roughly 600 words a page is 13,200 words, or about 17,600 tokens at three-quarters of a word per token. Split into 500 token windows with no overlap and you get 36 chunks, each of which begins and ends at a position determined by nothing except the count.

Now ask what happens to an answer. Suppose the sentence that answers a question, plus the context needed to make sense of it, spans 200 tokens. With non-overlapping 500 token windows, that span sits entirely inside one chunk only if it starts in the first 300 tokens of its window. That is 300 positions out of every 500, so 60 per cent. Two answers in five get cut across a boundary, and each half retrieves poorly because each half is missing the part that makes it relevant.

Add overlap. Keep the 500 token window and move it forward 400 tokens at a time. The span of 200 is now fully contained whenever it starts within the first 300 tokens of any window's 400 token stride, which is 300 out of 400, or 75 per cent. Chunk count rises from 36 to 44, about 22 per cent more storage and 22 per cent more embedding cost, to move intact retrieval from three fifths to three quarters.

That trade is almost always worth taking, and it is the cheapest single improvement available in a retrieval build. Scale it up and the absolute cost stays small. Four hundred documents averaging 18 pages is 4.32 million words, or 5.76 million tokens, which gives 11,520 chunks at a 500 token stride and 14,400 at a 400 token stride. At 1,536 dimensions in four-byte floats, 6,144 bytes a vector, that is about 71 megabytes against 88. Seventeen megabytes is not a budget conversation.

The same arithmetic also tells you where overlap stops helping. If your answer spans run 400 tokens rather than 200, a 500 token window contains them only 20 per cent of the time with no overlap, and even a 400 token stride only lifts that to 25 per cent. At that point the window is too small for the content and overlap is patching the wrong thing. Measure a handful of real answer spans in your own documents before choosing a window size, because a legal clause and a paragraph of exhibitor feedback are not the same length.

Splitting on headings, and keeping the path

A post-show report divides at its headings because the author already did the work of grouping related material. Use it.

Split at every heading. Where a section runs longer than your size cap, split inside it with overlap as above. Then prefix every chunk with its heading path before embedding, so the text that gets vectorised begins with something like "Autumn Show 2025 post-show report > Exhibitor feedback > Rebooking intent" and continues into the body.

The prefix does two jobs. It gives the embedding some of the context a human reader gets from knowing where they are in a document, and it gives the answer a citation string for free.

It also fixes a failure that is otherwise hard to diagnose. Every post-show report in a portfolio has a section called "Attendance" and a section called "Exhibitor feedback", so the bodies of those sections across eight shows and five years are semantically near-identical. Without the path prefix, a question about the 2025 kitchen and bath show will happily retrieve the attendance section of the 2022 hospitality show, and the answer will be fluent, sourced and about the wrong event. Prefixing the show name and edition into the embedded text pulls those forty near-duplicates apart.

Contracts need the opposite treatment. Splitting a contract at clause boundaries produces chunks that are individually meaningless, because the defined terms live in clause 1 and the obligations that use them live twenty pages later. For that document type, either keep the definitions block prepended to every clause chunk, which costs storage and works, or accept that contract questions need the whole document in context and route them differently. Both are defensible. Pretending clause 14 is self-contained is not.

Is a smaller unit always better?

Not always, and the published work points in an interesting direction here.

Chen and colleagues asked in 2023 what retrieval granularity to use, comparing passages, sentences and what they call propositions, defined as atomic expressions each carrying a distinct factoid. Their finding was that indexing by fine-grained units such as propositions outperformed passage-level units on retrieval, and improved downstream question answering within a fixed context budget.

The mechanism is worth understanding rather than copying. A large chunk contains the answer plus a lot of unrelated material, and the embedding averages all of it, so the chunk's vector drifts away from the specific question. A small unit has a sharp vector and no context.

For event documents the practical compromise is to retrieve small and read large. Index at the sentence or short-paragraph level for matching, then expand each hit to its surrounding section before handing anything to the model. You get the precision of the small unit in ranking and the context of the large one in generation, and it costs one extra lookup.

Where should the retrieved chunks sit in the prompt?

Not in the order the retriever returned them.

Liu and colleagues published the finding in Transactions of the Association for Computational Linguistics, volume 12, in 2024. Testing multi-document question answering and key-value retrieval, they found performance "is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models".

The consequence for a chunking strategy is direct. If your retriever returns 12 passages and you paste them in ranked order, your second and third best passages land in the worst position in the prompt.

Two responses, both cheap. Return fewer passages, because a middle that is two passages long has no middle to get lost in. And order them so the strongest sits first and the next strongest sits last, filling inward. If you are retrieving 12 passages because recall at 5 is poor, the fix belongs upstream in the retriever rather than in the prompt.

Testing two strategies on the same thirty questions

The comparison is straightforward and most teams skip it because it requires deciding what the right answer is.

Take 30 real questions about your archive. For each, open the document by hand and record which section contains the answer. That is an afternoon and it is the only part that cannot be automated.

Then index the same corpus twice, once with 500 token fixed chunks and once with heading-based chunks, and for each question record whether a chunk from the correct section appears in the top five. The score is a count out of 30 for each strategy. Where they disagree, read the pair: the fixed chunker's failures will cluster around section boundaries and tables, and the heading chunker's failures will cluster in long unstructured sections and in documents whose headings are inconsistent.

Keep the 30 questions. They become the seed of the retrieval score you run against every later change, which needs its own treatment separate from the answer the model writes, covered in measuring retrieval quality.

The limit

Chunking cannot fix a document that was never written to be read out of order.

A post-show report that says "as noted above, this was largely driven by the co-located event" is unusable as an isolated chunk, and pronouns referring across sections are everywhere in internal documents. Some of that can be repaired at index time by rewriting each chunk to resolve its references, which costs a model call per chunk and introduces a new place for errors to enter.

The deeper limit is that tables and figures, which is where an event team keeps most of its facts, do not chunk at all. A registration-by-channel table split across a boundary produces two chunks of numbers with no column headers on the second. Extract tables separately, store them as structured rows, and answer questions about them by querying rather than retrieving. That boundary between prose retrieval and structured query is where most of the design effort belongs, and it sits alongside the decision to run a keyword index next to the vector one for stand numbers and product codes. The wider set of choices lives on the AI platform side, and the retriever and generator pairing itself is covered in retrieval augmented generation over your own documents.

Start by taking one post-show report and one signed contract, running both through whatever splitter you have, and reading ten chunks from each at random. If more than one or two of the contract chunks would be meaningless to a colleague shown them cold, your splitter is wrong for that document type and no amount of embedding quality will cover it.