Why hybrid keyword and vector search beats embeddings alone on event documents

Someone types "who was on stand 4C12 in 2024" into the assistant and gets back a confident paragraph about stand 4C13. Somebody else searches for the SIC code the sales team uses to segment machine tool buyers and gets a general answer about manufacturing exhibitors. Neither is a model failure. Both are the vector index doing exactly what a vector index does, which is why hybrid keyword and vector search is the sensible default for an event document archive rather than an optimisation to consider later.

The tokens that matter most in this business are the ones with the least meaning.

The questions that break an embedding index

An embedding compresses a passage into a fixed-length vector representing what it is about. That compression is the whole point and it is lossy in a specific direction: rare surface forms carrying no semantic content get averaged away by the surrounding text.

Look at what an event corpus is full of. Stand numbers, 4C12 and 7A03. Hall identifiers, 9.1 and North 3. Badge and order references. Exhibitor SIC or NAICS codes. Show acronyms that mean nothing outside the building. Product part numbers in exhibitor profile forms. Contract clause numbers. Named individuals with common surnames.

A passage mentioning 4C12 embeds to almost the same place as a passage mentioning 4C13, because to the model the two strings differ by one character in a region of the token space where no meaning lives. Semantic similarity is the wrong tool for a question whose answer depends on exact identity.

Why does BM25 still win on a stand number?

Because a keyword scorer weights a term by how rare it is, and rarity is precisely the property that makes 4C12 useful.

Work the inverse document frequency by hand on the 5,760-passage index from a mid-sized portfolio archive. The BM25 form of IDF for a term appearing in n of N documents is the natural log of N minus n plus 0.5, over n plus 0.5.

Take 4C12, appearing in 3 passages out of 5,760. That is the log of 5,760 minus 3 plus 0.5, which is 5,757.5, over 3.5. The log of 1,645 is 7.41.

Now take the word "exhibitor", appearing in 4,600 of the 5,760. That is the log of 1,160.5 over 4,600.5, which is the log of 0.252, or minus 1.38. Negative, and implementations floor it at or near zero.

So on a query reading "exhibitor on stand 4C12", the token 4C12 contributes essentially all of the score and the word "exhibitor" contributes nothing. The scorer has correctly worked out which word in the query identifies the answer. An embedding of that same query produces one vector in which both words have already been mixed together, and the mixing cannot be undone at search time.

What the published comparison actually found

Thakur and colleagues built BEIR for the NeurIPS 2021 Datasets and Benchmarks track, assembling 18 publicly available retrieval datasets across diverse tasks and domains and evaluating 10 retrieval systems spanning lexical, sparse, dense, late-interaction and re-ranking approaches. Their conclusion was that BM25 is "a robust baseline", that re-ranking and late-interaction models achieved the best zero-shot performance at high computational cost, and that dense and sparse retrievers were cheaper and "often underperform other approaches".

Set that against Karpukhin and colleagues at EMNLP 2020, who reported their dense passage retriever beating a strong Lucene BM25 system by 9 to 19 points absolute on top-20 retrieval accuracy. Both results are correct and the difference between them is the whole argument.

Dense retrieval wins when it has been trained on questions from the domain it is searching. BEIR measures the zero-shot case, where it has not. Your internal archive of post-show reports and sponsorship agreements is zero-shot for every embedding model you can buy, and nobody is going to fine-tune one on 400 documents.

That is the honest reason to run both indexes. The case where dense retrieval clearly wins is the case you are not in, and hedging against that has a better claim on your effort than picking a favourite retriever.

The cost of running both is one extra index over the same passages. Keyword indexing is cheap enough that the storage barely registers next to the vectors, and the extra query latency is a few milliseconds against a vector scan that was already fast. There is no version of this decision where the second index is the expensive part.

Combining two ranked lists without a tuning project

The obvious approach is to normalise both scores and take a weighted sum, and it is a trap, because BM25 scores and cosine similarities live on incomparable scales that shift with query length and corpus composition. You end up maintaining a weight that has to be retuned whenever the corpus grows.

Rank fusion sidesteps it. Score each passage as the sum, over retrievers, of one divided by a constant plus that passage's rank in that retriever's list. Sixty is the constant in common use.

Work two cases. A passage ranked 1 by the keyword index and 30 by the vector index scores one over 61 plus one over 90, which is 0.01639 plus 0.01111, or 0.02750. A passage ranked 4 by both scores one over 64 twice, which is 0.03125.

The passage both retrievers liked reasonably beats the passage only one retriever loved. That is the behaviour you want on an event archive, where a single retriever ranking something first is often a sign it has locked onto a surface coincidence. Nothing needs tuning and nothing breaks when the corpus doubles.

The constant is worth understanding rather than copying. It flattens the difference between the top few ranks: with 60 in the denominator, rank 1 scores 0.01639 and rank 5 scores 0.01538, a gap of 6 per cent, while the gap between rank 1 and rank 50 is only about 45 per cent. A smaller constant sharpens the top of the list and makes a single retriever's first place harder to overturn. If you want the keyword index to dominate on exact identifier matches, lowering the constant is the lever, and it is one number rather than a weighting scheme.

One caveat on fusion generally: it can only reorder what was retrieved. If the correct passage sits at rank 300 in both lists and you fuse the top 50, no constant recovers it. Fusion is a merging rule and the recall problem stays upstream.

Which fields belong in the keyword index?

More than the body text, and this is where most of the available improvement sits.

• Structured identifiers extracted at index time, including stand numbers, hall codes, order references and contract numbers, stored as their own field rather than left inline.
• The show name and edition year, so a filter on them is exact rather than semantic.
• Exhibitor and company names, normalised for legal suffixes, since a company name is a rare token and an embedding treats it as a topic.
• Document type, so "find the contract" narrows before ranking begins.

Extracting those is a regular expression job for the identifier formats your business actually uses, and it takes an afternoon per format. The gain is that a query containing 4C12 can be answered by a filter followed by a rank, which is faster and more accurate than either retriever alone.

Curating synonyms into the keyword index is the other half of this, and it is unglamorous work that pays. Every organiser has an internal vocabulary that appears nowhere in a general embedding model: rebook and renew and resign used interchangeably, net square feet and NSF and net square metres, hosted buyer and VIP buyer and qualified buyer. A synonym list of forty entries, maintained by the person who owns the post-show report, does more for retrieval on an event archive than a larger embedding model does. Keep it in version control with a note on who added each entry and why, because the list is a piece of business knowledge and it will outlast the retrieval stack it was written for.

Where hybrid does not help

It does nothing for a question whose answer requires a synonym the author never wrote. If the report says "rebooking intent" and the question asks about "renewal likelihood", the vector index carries that and the keyword index contributes noise. Hybrid retrieval is additive on the identifier questions and roughly neutral on the paraphrase questions, which is a good trade and not a free one.

It also does nothing about a passage that was never created as a unit, which is the upstream decision covered in picking a chunking strategy, and it does nothing about numbers that live in the warehouse rather than in prose, which is one of the boundaries drawn in retrieval augmented generation over your own documents.

The limit

Running two indexes doubles the number of things that can be silently misconfigured, and the failure is quiet. If your keyword analyser lowercases and strips punctuation, 4C12 may survive while 4-C-12 becomes three tokens, and the two forms appear in different documents from different years. If your tokeniser splits on digit boundaries, the stand number stops being a rare term at all and the entire advantage disappears without any error appearing anywhere.

The only defence is to test the identifiers directly. Take twenty real stand numbers, twenty exhibitor names and ten contract references from your own archive, query each one, and check that the document you know contains it comes back at rank one. That test takes an hour, it will fail the first time, and what it fails on is almost always the analyser configuration rather than the retrieval model. Turning that into an ongoing score is the job described in measuring retrieval quality, and the wider set of choices sits on the AI platform side.

Start with the twenty stand numbers. If more than two of them fail to return their own document at rank one, adding a keyword index is worth more this week than any change to the embedding model.