Hybrid search in RAG: dense and sparse

What is hybrid search?

Hybrid search runs two kinds of retrieval on the same query and merges the results: a sparse search that matches the words in the query, and a dense search that matches its meaning.

Real questions need both. A legal team may search for the exact phrase "force majeure" and also want passages about events beyond a party's control. A support team may search for an exact error code and also for a plain-language description of the fault. Each method covers the other's blind spot.

What do sparse embeddings capture?

Exact terms. A sparse vector has one position per vocabulary term, and nearly all positions are zero. The classic scoring function is BM25, which weighs how often a term appears in a document, how rare it is across the collection, and the document's length. BM25 is the default similarity in Elasticsearch, with its term-saturation parameter k1 set to 1.2 and its length-normalization parameter b set to 0.75 by default.^[1]

Strengths:

• Rare, specific terms such as product codes, clause numbers and internal acronyms score highly.
• Results are explainable: you can see which terms matched.
• New jargon matches as soon as it is in the index.

The blind spot is vocabulary mismatch. Classic keyword search will not equate "cell phone" with "mobile phone" unless both appear. Learned sparse models such as SPLADE narrow this gap by expanding terms while keeping the vectors sparse, so they can still use an inverted index.^[2]

What do dense embeddings capture?

Meaning. A dense embedding is a list of numbers produced by a model, and "the distance between two vectors measures their relatedness."^[3] OpenAI's text-embedding-3-small, for example, returns vectors of 1,536 dimensions by default.^[3]

Strengths:

• Synonyms and paraphrases match without hand-built synonym lists.
• Long, descriptive questions find relevant passages that share few exact words.

The blind spot is exact matching. A query for a specific standard or part number can return passages about the general topic and miss the one document that names it. Dense models can also struggle outside the domain they were trained on. The BEIR benchmark, which tests retrieval models on unseen datasets, found BM25 a strong baseline and pointed to "considerable room for improvement" in how dense retrievers generalize.^[4]

How does hybrid search fuse the two lists?

The two scores live on different scales, so a raw BM25 score cannot be compared directly with a cosine similarity. Three common fusion methods:

• Reciprocal rank fusion (RRF). Ignore the scores and use positions. Each list adds one divided by a constant plus the document's rank; the sums give the final order. Elasticsearch's RRF sets that constant to 60 by default and states that "RRF requires no tuning."^[5]
• Weighted score fusion. Normalize both scores to the same range, then combine them with a weight that favors keywords or meaning. It gives more control, but the weight must be tuned on your own test questions.
• Reranking. Send the merged candidates to a cross-encoder that reads the query and each passage together and scores relevance directly. The Sentence Transformers documentation describes this retrieve-then-rerank pattern and notes the cost of scoring many pairs this way.^[6]

RRF is a sensible default to start with. Our guide to chunking and reranking for advanced RAG covers the reranking stage in detail.

Where does hybrid search help most?

Where queries mix exact identifiers with natural language:

• Internal knowledge bases, where staff search for policy IDs, error codes or acronyms as well as plain questions.
• Product and parts catalogs, where part numbers must match exactly but descriptive searches must still find the right category.
• Legal and regulatory text, where citations must match verbatim but guidance is often paraphrased.
• RAG grounding, where the model needs the exact passage that holds a fact and the passages that explain it.

When is one method enough?

Sometimes. A short, structured FAQ searched by exact terms may need only keyword search. A collection of long articles searched with descriptive questions may do well with embeddings alone. Test each method on its own first, and add the second only if it finds answers the first misses.

What does hybrid search cost to run?

Two indexes and two lookups per query. You keep an inverted index for sparse search and a vector index for dense search, usually an approximate nearest neighbor structure such as HNSW, a graph-based method for approximate K-nearest neighbor search.^[7] At query time you run both searches, fuse the lists and, optionally, rerank.

That adds latency, storage and pipeline work: both indexes must be updated and deleted in step when documents change. Our guide to enterprise RAG architecture covers keeping an index fresh, and Pinecone vs pgvector compares two common vector stores.

How do you tune and test it?

Treat fusion settings as parameters to measure, not to set once:

1. Write a test set of real questions with the passages that answer them, including exact-term and descriptive questions.
2. Measure recall for sparse only, dense only and fused results.
3. If you use weighted fusion, try several weights and keep the one that serves both question types.
4. Add a reranker and check whether answer quality gains justify the extra latency.

Clean input matters as much as fusion; see our guide to normalizing unstructured data for LLMs. Our data infrastructure for AI service covers these steps as part of our AI consulting work.