Retrieval augmented generation for organisers working from their own documents

Somebody asks what the team promised the hall two sponsor last year. The answer is in a signed agreement, and the person who negotiated it left in March. Somebody else asks why the 2024 edition moved from three days to two, and the reasoning is in a board paper that four people have read. Retrieval augmented generation for organisers is worth taking seriously because this archive exists in every portfolio business, none of it is on the public web, and no model was ever trained on any of it.

The corpus is prior post-show reports, event briefs, sponsorship agreements, floor plan sign-offs, exhibitor feedback summaries and board packs. It is the institutional memory of the business, and today it is searched by opening SharePoint and guessing at filenames.

What the method actually adds

Lewis and colleagues introduced retrieval augmented generation at NeurIPS in 2020, combining what they called parametric memory, the knowledge sitting in a pre-trained sequence-to-sequence model's weights, with non-parametric memory, a dense vector index reached through a neural retriever. They defined two forms, one that conditions the whole generated sequence on a single retrieved set and one that can draw on different passages for each token.

The architectural point that survives translation to an event business is simple enough. The model supplies fluency and the ability to read a question. The index supplies the facts. Change the index and the answers change, with no retraining, which is the property that makes the approach usable for a document set that grows by forty files after every show.

Karpukhin and colleagues had shown at EMNLP the same year that a learned dense retriever could beat a strong Lucene BM25 system by 9 to 19 points absolute on top-20 passage retrieval accuracy across open-domain question answering datasets. That result is the reason vector search became the default. It was also measured in domain, on questions the retriever had been trained for, and the out-of-domain picture is different enough to need a keyword index running alongside the vector one.

What does a grounded answer have to carry?

A file name and a page number, at minimum, for every claim.

This is harder than it looks and it is measurable. Gao and colleagues built the ALCE benchmark for EMNLP 2023 specifically to score citation quality alongside fluency and correctness, and their result deserves reading twice: on the ELI5 dataset, even the best systems they tested lacked complete citation support 50 per cent of the time. Half the sentences that carried a citation were not fully supported by the thing they cited.

So a system that prints sources next to its answers is not thereby trustworthy. It is trustworthy when someone has checked how often the source says what the sentence says, which is a measurement you have to run on your own corpus rather than inherit.

The design consequence is to make checking cheap. Render the retrieved passage next to the sentence, not behind a link, with the page number and the document's last-modified date. A show director will spot check three of them and stop, and three spot checks against visible passages catch more than zero spot checks against a footnote.

Sizing the index for a portfolio archive

The arithmetic is worth doing because it reframes what the project is.

Take 400 documents averaging 18 pages at roughly 600 words a page. That is 400 times 18 times 600, or 4,320,000 words. Split into passages of about 750 words and you have 5,760 passages. Embed each into a 1,536-dimension vector stored as four-byte floats: 1,536 times 4 is 6,144 bytes per vector, and 5,760 times 6,144 is 35.4 megabytes.

Thirty-five megabytes. The entire searchable memory of an eight-show portfolio fits in a file smaller than one exhibitor's stand renders, and a brute force scan over it returns in milliseconds without any index structure at all.

Which means none of the difficulty is in the vector store, and any vendor conversation that centres on it is a conversation about the wrong thing. The difficulty is in the four questions underneath: which documents belong in the corpus, how they get split, who may retrieve each one, and how you know the retrieval worked.

One question, traced end to end

Take the sponsor question and follow it through the 5,760 passages.

The question is "what did we promise the hall two headline sponsor for the 2025 edition". The retriever returns five passages. Two come from the signed sponsorship agreement, one from a pre-show internal briefing, one from a post-show reconciliation note, and one from a 2023 agreement with a different sponsor that ranked because the language is boilerplate.

Three of those five are useful, one is noise, and one, the 2023 agreement, is actively dangerous, because it reads exactly like the right answer and describes a different deal. A model handed all five without instruction will blend them.

So the prompt has to do two things. It has to require that every claim names the document and page it came from, and it has to permit an answer that uses only some of the passages. The output you want reads like this: the agreement committed 4 branded aisle banners, 2 speaking slots and a 6 by 4 metre stand, citing the sponsorship agreement pages 3 and 4, and the reconciliation note records that only 3 banners were installed, citing page 1.

Now the show director can check two documents in about ninety seconds. Whether the retriever found the right passages in the first place is a separate score with its own method, covered in measuring retrieval quality on its own, and it is the score that fails first.

Who is allowed to retrieve which document?

This is the question that stops most internal deployments, correctly.

A board pack contains commercially sensitive material about acquisitions and staff. A sponsorship agreement contains rates that other sponsors are not shown. An exhibitor feedback summary contains named criticism of a named account manager. A single index over all of it, queried by anyone with a login, is a data breach with a chat interface.

The workable pattern is to attach an access label to every passage at index time, carry the querying user's entitlements into the retrieval call, and filter before ranking rather than after. Filtering after ranking is the mistake that keeps recurring: it produces a top five that silently loses two results, and the user has no idea their answer was assembled from partial evidence.

There is a second-order leak worth naming. Even with correct filtering, an answer that says "no relevant documents were found" for one user and returns a detailed answer for another reveals that a document exists. For most event content that is acceptable. For anything touching an acquisition it is not, and those documents belong in a separate corpus with its own access list rather than a flag on a shared one.

Where retrieval is the wrong mechanism

Numbers.

A post-show report states that registration closed at 14,382. That figure was true when the report was written. It may have been a draft, it may exclude late walk-ups, and the warehouse may now hold 14,551 after a deduplication pass. Retrieval will find the sentence, the model will repeat the number, and the citation will be perfectly accurate about a document that is out of date.

Route numeric questions to the warehouse and keep retrieval for the questions whose answers only exist in prose: what was agreed, what was decided, what the reasoning was, what went wrong. A router in front of both is more useful than a better retriever, and the split between them is the substance of when a stored query beats a generated one.

I would go further. If your first retrieval build indexes the post-show PDFs because they are easy to find, you have built a system whose most confident answers are its most stale. Index the decisions first: board papers, contract terms, incident notes, the email chain where the hall change was agreed. Those never go out of date, because they record what happened at a moment rather than a running total.

Where this stops

The corpus is a sample of what the organisation knows, and the sampling is not random. Documents get written when something needs signing off or reporting upward. The reason a show underperformed in 2023 may exist only in a conversation, and a retrieval system will answer the question anyway, from whatever adjacent document ranks highest, with a citation attached.

Absence is also invisible. A question whose answer is genuinely not in the corpus should return nothing, and a generative system will usually produce something. Test for that deliberately: write ten questions you know the archive cannot answer, run them, and count how many come back with a confident answer and a plausible citation. If the count is above one, the abstention behaviour needs fixing before anything else does. How the documents were split into passages changes that count more than most people expect, which is the subject of picking a chunking strategy, and the wider set of controls sits on the AI platform side.

Start by listing 20 questions your team has actually asked about a past edition in the last month, then find the document that answers each one by hand and note how long it took. That list is your first evaluation set, and the median time to find each answer is the baseline any retrieval build has to beat.