Hybrid Search and Re-ranking for RAG: BM25, RRF, Cross-Encoders

A company in Indore gets 800 applications for one data analyst post. HR first runs two quick screens. One filters CVs that contain the exact words "SQL" and "Power BI". The other has a recruiter skim for people who clearly did analytics work, even if they wrote "built dashboards" instead of "Power BI".

Each screen misses people the other catches, so HR merges both shortlists. Then a senior analyst spends real time interviewing the top 30 and picks the final 5.

That is hybrid search followed by re-ranking. Two cheap, different screens find candidates; one expensive, careful judge orders them.

What is hybrid search?

Mapping the hiring: the exact-words filter is keyword search (BM25), the recruiter skim is vector (dense) search, merging the shortlists is rank fusion, and the interview is the re-ranker.

Hybrid search is a retrieval method that runs a keyword (sparse) search and a vector (dense) search for the same query and combines their results into one ranking, so that both exact term matches and meaning-based matches are found.

What is RAG? used TF-IDF, a keyword method. Vector Databases Explained covered dense search. This lesson combines them on the same fictional Kshipra Institute of Technology (KIT), Indore handbook, chunked as in RAG chunking strategies.

Why vector search alone is not enough

Dense embeddings are great at paraphrase but weak at rare exact tokens: codes, IDs, abbreviations and names the model barely saw in training.

Real users ask both kinds, so hybrid search stops you choosing.

How BM25 keyword scoring works

BM25 (Best Matching 25) is the standard keyword ranking function in search engines such as Elasticsearch, OpenSearch and Lucene. It improves on TF-IDF in two ways:

1. Term frequency saturates. The tenth "refund" in a chunk adds far less than the first, controlled by k1 (usually 1.2 to 2.0).
2. Length is normalised. A long chunk does not win just by containing more words, controlled by b (usually 0.75).

Each query term contributes idf × tf × (k1 + 1) / (tf + k1 × (1 − b + b × length / avg_length)). Here it is written by hand on seven KIT chunks.

import re
import numpy as np

chunks = {
    "exams#atkt":    "ATKT supplementary exams for failed subjects are held in July; the ATKT form closes on 20 June.",
    "exams#reval":   "Revaluation of an exam paper costs 300 rupees and must be requested within 7 days of the result.",
    "fees#late":     "Semester fees are due by 15 July and a late fee of 100 rupees per day applies.",
    "fees#refund":   "If you withdraw before classes start, the full fee is refunded minus 1,000 rupees.",
    "fees#credit":   "Approved refunds are credited to your bank account within 10 working days.",
    "hostel#curfew": "Hostel residents must be inside by 10 pm; late entry needs warden approval.",
    "library#fine":  "Late return of library books costs 2 rupees per day per book.",
}
STOP = set("the a of and for are is in on to by my if you be per i when what will after".split())
tok = lambda t: [w for w in re.findall(r"[a-z0-9]+", t.lower()) if w not in STOP]
docs = [tok(t) for t in chunks.values()]
N, avgdl = len(docs), np.mean([len(d) for d in docs])

def bm25(query, k1=1.5, b=0.75):
    scores = np.zeros(N)
    for term in set(tok(query)):
        df = sum(term in d for d in docs)
        idf = np.log(1 + (N - df + 0.5) / (df + 0.5))
        for i, d in enumerate(docs):
            tf = d.count(term)
            scores[i] += idf * tf * (k1 + 1) / (tf + k1 * (1 - b + b * len(d) / avgdl))
    return scores

ids = list(chunks)
for q in ["ATKT form last date", "When will my money come back after I leave?"]:
    s = bm25(q)
    print(q, "->", [(ids[i], round(float(s[i]), 2)) for i in np.argsort(-s)[:3]])

For "ATKT form last date", BM25 puts the ATKT chunk on top and every other chunk scores 0. For the refund question every score is 0: no chunk shares a word with "money come back". That blind spot is exactly what dense search fixes.

Fusing rankings with Reciprocal Rank Fusion

You now have two ranked lists per question. You cannot simply add their scores: BM25 scores are unbounded (3.93 above), while cosine similarities sit between -1 and 1.

Reciprocal Rank Fusion (RRF) sidesteps the scale problem by using only positions. Each document gets 1 / (k + rank) from every list it appears in, summed. The constant k, usually 60, stops the very top rank from dominating.

Two cheap retrievers run in parallel, RRF merges their lists, and a cross-encoder re-ranks a short candidate list before the LLM sees anything.

The dense rankings below stand in for what an embedding model would return; the fusion logic is exactly what production systems run.

# Rankings from two retrievers for the same questions (best first).
runs = {
    "ATKT form last date": {
        "bm25":  ["exams#atkt"],                                  # only exact match
        "dense": ["exams#reval", "fees#late", "exams#atkt", "attendance#min"],
    },
    "When will my money come back after I leave?": {
        "bm25":  [],                                              # no shared words
        "dense": ["fees#credit", "fees#refund", "fees#late", "library#fine"],
    },
    "late fee for semester fees": {
        "bm25":  ["fees#late", "library#fine", "hostel#curfew"],
        "dense": ["fees#late", "fees#refund", "library#fine"],
    },
}

def rrf(rankings, k=60):
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores.items(), key=lambda kv: -kv[1])

for question, r in runs.items():
    fused = rrf([r["bm25"], r["dense"]])
    print(question)
    print("   fused top-3:", [(d, round(s, 4)) for d, s in fused[:3]])

All three questions now come out right. The ATKT chunk wins because it appears in both lists. The refund question is carried by dense search alone. For "late fee for semester fees", both retrievers agree on fees#late, and library#fine takes second place because it also appears in both lists, while fees#refund appears in only one.

That last point is RRF's real strength: agreement between retrievers is rewarded, without anyone tuning score scales.

Weighted score fusion

The alternative is to normalise each score list to 0 to 1 and mix them with a weight, often called alpha. Several vector databases expose this knob directly.

import numpy as np

# Raw scores live on different scales, so normalise before mixing them.
ids   = ["exams#atkt", "exams#reval", "fees#late", "attendance#min"]
bm25  = np.array([7.9, 0.0, 0.0, 0.4])        # unbounded keyword scores
dense = np.array([0.61, 0.74, 0.66, 0.58])    # cosine similarities

def minmax(x):
    return (x - x.min()) / (x.max() - x.min() + 1e-9)

for alpha in [0.0, 0.3, 0.5, 1.0]:            # alpha = weight on dense
    mixed = alpha * minmax(dense) + (1 - alpha) * minmax(bm25)
    order = np.argsort(-mixed)
    print(f"alpha={alpha}: " + ", ".join(f"{ids[i]}={mixed[i]:.2f}" for i in order[:2]))

With alpha at 0.3 or 0.5 the ATKT chunk stays first, ahead of revaluation. At 1.0 (pure dense) revaluation wins and the right answer is lost. Weighted fusion can beat RRF when you tune alpha on real questions, but min-max normalisation is fragile: one outlier score squashes everything else. RRF is the safer default.

What is re-ranking in RAG?

Retrievers are built for speed over millions of chunks. They encode the question and each chunk separately (a bi-encoder), so chunk vectors can be computed in advance. The price is that the model never sees question and chunk side by side.

Re-ranking is a second retrieval stage in which a more accurate but slower model scores each (question, candidate chunk) pair together and reorders a short candidate list, keeping only the best few.

The usual re-ranker is a cross-encoder: a transformer that reads the question and the chunk as one input and outputs a relevance score. Because it attends across both texts, it can tell "this chunk mentions refunds" from "this chunk answers how long a refund takes".

Bi-encoders compare precomputed vectors, so they scale to millions of chunks. Cross-encoders read each pair together: more accurate, but only affordable on a short list.

The toy below imitates a cross-encoder with hand-written rules: it checks whether the chunk meets every need in the question. A real cross-encoder learns these judgements from data.

import re

question = "How many days does a refund take?"
# First-stage candidates from hybrid search, in fused order
candidates = {
    "fees#refund":  "If you withdraw before classes start, the full fee is refunded minus 1,000 rupees.",
    "exams#reval":  "Revaluation of an exam paper costs 300 rupees and must be requested within 7 days.",
    "fees#credit":  "Approved refunds are credited to your bank account within 10 working days.",
    "library#fine": "Late return of library books costs 2 rupees per day per book.",
}
# Toy stand-in for a cross-encoder: it reads question and chunk together and
# checks that every need in the question is met, using knowledge of related words.
needs = {
    "refund":   {"refund", "refunds", "refunded"},
    "duration": {"days", "day", "weeks"},
    "timing":   {"within", "credited", "processed"},
}

def rerank_score(chunk):
    words = set(re.findall(r"[a-z]+", chunk.lower()))
    return sum(bool(words & variants) for variants in needs.values()) / len(needs)

reranked = sorted(candidates, key=lambda c: -rerank_score(candidates[c]))
print("before:", list(candidates))
print("after :", [(c, round(rerank_score(candidates[c]), 2)) for c in reranked])

pairs, ms_per_pair = 30, 12     # assumed cost per pair; measure on your own hardware
print(f"Re-ranking {pairs} candidates adds about {pairs * ms_per_pair} ms per question")

Before re-ranking, the chunk that merely mentions refunds sits first. After it, fees#credit, the only chunk that says how long a refund takes, rises to the top. The last line is the cost reminder: a re-ranker runs once per candidate, so you re-rank 20 to 50 chunks, never the whole index.

A real cross-encoder with sentence-transformers

This uses the sentence-transformers CrossEncoder API and a small public MS MARCO re-ranking model. Install with pip install sentence-transformers; it downloads the model, so it cannot run in the browser.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")

question = "How many days does a refund take?"
candidates = [
    "If you withdraw before classes start, the full fee is refunded minus 1,000 rupees.",
    "Revaluation of an exam paper costs 300 rupees and must be requested within 7 days.",
    "Approved refunds are credited to your bank account within 10 working days.",
]

for hit in reranker.rank(question, candidates, top_k=2):
    print(f"{hit['score']:.2f}  {candidates[hit['corpus_id']]}")

These MS MARCO models return raw logits, so scores can be negative; only the order matters. If users ask in Hindi or Hinglish, test a multilingual re-ranker instead.

Putting it together: a production recipe

1. Index twice: store each chunk's embedding and its BM25 terms. Elasticsearch, OpenSearch, Weaviate, Qdrant and PostgreSQL (pgvector plus full-text search) can all hold both.
2. Retrieve wide: take the top 20 to 30 from each retriever.
3. Fuse with RRF (k = 60) into one list of about 30 unique chunks.
4. Re-rank those with a cross-encoder and keep the top 3 to 5.
5. Threshold: if even the best re-ranked score is low, send no context and let the model say it does not know.
6. Measure each stage separately: did the right chunk reach the top 30, and did re-ranking move it into the top 5?

Common mistakes with hybrid search and re-ranking

• Adding raw scores. BM25 and cosine scores are on different scales. Use RRF or normalise first.
• Re-ranking too few candidates. If the right chunk is not in the first-stage list, the re-ranker cannot rescue it. Retrieve wider.
• Re-ranking too many. Cost grows with every pair. Watch latency at 100+ candidates.
• Skipping tokenisation details. BM25 depends on lower-casing, stop words and stemming. "Refund" and "refunds" are different tokens unless you stem.
• Tuning on gut feel. Choose alpha, k and candidate counts on a labelled question set, as covered in evaluating RAG systems.

When good retrieval still is not enough, for multi-part or vague questions, advanced RAG architectures add query rewriting, routing and self-correction on top.

Key takeaways

• Keyword search finds exact terms; vector search finds meaning. Hybrid search runs both.
• RRF merges ranked lists by position, rewards agreement and needs no score normalisation.
• Weighted fusion with alpha can work, but needs normalised scores and tuning.
• Cross-encoder re-ranking reads question and chunk together and fixes subtle ordering errors.
• Retrieve wide (about 30), re-rank narrow (top 3 to 5), and measure each stage.

If you would rather build this pipeline with feedback from a mentor, a mentor-led RAG and GenAI course in Indore at Debugshala covers retrieval projects in class. To keep your logic sharp, try sorting and ranking puzzles on Debugshala Test.