HyDE in RAG: Hypothetical Document Embeddings Explained

You lost your bag at Indore railway station and go to the lost-and-found counter. If you say "I lost something", the clerk can only shrug. If you say "a black backpack with a college sticker, a steel bottle in the side pocket", the clerk walks straight to the right shelf.

Your description need not be perfect. Maybe the bottle was aluminium, not steel. It still sounds like the things on the shelf, so it leads to the right one.

HyDE gives your retriever that description. Instead of searching with the bare question, it searches with a detailed, imagined answer.

What is HyDE in RAG?

Mapping the station: the vague request is the user's question, the detailed description is the hypothetical document an LLM writes, the shelf of bags is your vector index, and the clerk is nearest neighbour search. A wrong detail does not matter much, because the match is on overall resemblance.

HyDE (Hypothetical Document Embeddings) is a query transformation technique in which an LLM generates a hypothetical answer passage for the user's question, and that passage's embedding, rather than the question's, is used to retrieve real documents.

HyDE was introduced in the 2022 paper "Precise Zero-Shot Dense Retrieval without Relevance Labels" by Gao and colleagues. It is one of the pre-retrieval patterns listed in advanced RAG architectures, and it deserves its own lesson because it is easy to try and easy to misuse.

Why questions and documents don't match

Questions and answers are written differently. A student types "Can I get my money back if I quit after two weeks?". The KIT handbook (our fictional Indore college) says "Students who withdraw within the first 15 days of classes receive a 50 percent fee refund."

They share almost no words. "Money back" versus "refund", "quit" versus "withdraw", "two weeks" versus "15 days". Embedding models narrow this gap but do not close it, especially for short questions. This is called query-document asymmetry.

A generated answer is written as a statement in formal policy language, so it sits close to the real policy chunk in vector space, even if its numbers are invented.

How HyDE works, step by step

Traditional RAG embeds the question. HyDE inserts one LLM call that writes a hypothetical answer, then embeds and searches with that.

1. Generate: prompt an LLM with "Write a short passage from a college handbook that answers this question", plus the question.
2. Embed: embed the generated passage with the same embedding model used for your chunks. Optionally generate several passages and average their vectors with the question's vector.
3. Retrieve: run normal nearest neighbour search with that vector.
4. Answer: build the prompt from the real retrieved chunks and the original question. The hypothetical passage is thrown away.

Step 4 is the safety net. The invented passage only steers the search; it never appears as evidence.

HyDE in action on the KIT handbook

This toy uses TF-IDF so it runs in your browser. Keyword vectors exaggerate the mismatch, since they cannot link "money back" to "refund" at all, but the mechanism is the same with real embeddings.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

chunks = {
    "fees#partial":   "Students who withdraw within the first 15 days of classes receive a 50 percent fee refund.",
    "fees#full":      "Withdrawal before classes start earns a full refund minus a 1,000 rupee processing charge.",
    "fees#late":      "Semester fees are due by 15 July and a late fee of 100 rupees per day applies.",
    "hostel#leave":   "Hostel students going home for the weekend must sign the leave register.",
    "exams#reval":    "Revaluation of an exam paper costs 300 rupees per paper.",
    "attendance#med": "A medical certificate must reach the office within 7 days of rejoining.",
}
ids, texts = list(chunks), list(chunks.values())
vec = TfidfVectorizer(stop_words="english").fit(texts)
M = vec.transform(texts)

def ranking(text):
    s = cosine_similarity(vec.transform([text]), M)[0]
    return [(ids[i], round(float(s[i]), 2)) for i in s.argsort()[::-1][:3]]

question = "Can I get my money back if I quit after two weeks?"
# What an LLM might write when asked to "answer as the handbook would".
# The numbers are invented and partly wrong; that is fine for HyDE.
hypothetical = ("If students withdraw in the first few weeks of classes, the college "
                "returns part of the fee as a refund, often about 40 percent.")

print("Search with question    :", ranking(question))
print("Search with hypothetical:", ranking(hypothetical))

The question scores 0.0 against every chunk: no shared words at all. The hypothetical passage finds fees#partial with a score of 0.76, far ahead of the rest. It said "40 percent" where the real rule is 50, and it still led to the right chunk, because it used the handbook's words: withdraw, classes, fee, refund, percent.

When HyDE goes wrong, and how to anchor it

HyDE relies on the LLM knowing roughly what an answer looks like. On local jargon it may not. "ATKT" is a term many Indian universities use for supplementary exams after failing a subject. A general model may not connect it to the right idea and write about ordinary exam forms instead.

The fix is to keep the question in the search. The HyDE paper averaged the question's vector with the vectors of several generated passages. Here the question gets half the weight, a common tweak:

Each hypothetical answer lands somewhere near the real documents. Averaging several of them with the question cancels out individual mistakes.

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer

chunks = {
    "exams#atkt":  "ATKT supplementary exams for failed subjects are held in July; the ATKT form closes on 20 June.",
    "exams#reval": "Revaluation of an exam paper costs 300 rupees and must be requested within 7 days of the result.",
    "exams#hall":  "Hall tickets for end-semester exams are issued one week before the first paper.",
    "fees#late":   "Semester fees are due by 15 July and a late fee of 100 rupees per day applies.",
}
ids, texts = list(chunks), list(chunks.values())
vec = TfidfVectorizer(stop_words="english").fit(texts)
M = vec.transform(texts).toarray()
M /= np.linalg.norm(M, axis=1, keepdims=True)

def embed(t):
    v = vec.transform([t]).toarray()[0]
    return v / (np.linalg.norm(v) + 1e-9)

question = "What is the last date for the ATKT form?"
# Three generated "answers" from a model that does not know what ATKT means here
hypotheticals = [
    "The exam form must be submitted before the result, and late forms cost 300 rupees.",
    "Hall tickets are issued after the exam form is submitted one week before the paper.",
    "Repeat exam forms must be submitted by the June deadline.",
]
options = {
    "question only":            embed(question),
    "hypotheticals only":       np.mean([embed(h) for h in hypotheticals], axis=0),
    "question 50% + hypos 50%": 0.5 * embed(question) + 0.5 * np.mean([embed(h) for h in hypotheticals], axis=0),
}
for name, v in options.items():
    s = M @ v
    print(f"{name:<25}", [(ids[i], round(float(s[i]), 2)) for i in np.argsort(-s)[:2]])

The question alone finds the ATKT chunk, thanks to the exact term. The drifting hypotheticals alone rank revaluation first: HyDE has made things worse. Mixing the two puts the ATKT chunk back on top, 0.36 against 0.17.

That is the lesson of HyDE in one table:

HyDE with a real LLM and embedding model

This version uses the OpenAI Python SDK's Responses API to write the passage and a sentence-transformers model to embed it. It needs API keys and a model download, so it does not run in the browser. Swap in any chat model and the same embedding model you indexed with; see the OpenAI API quickstart and the sentence-transformers docs.

import numpy as np
from openai import OpenAI
from sentence_transformers import SentenceTransformer

client = OpenAI()                                   # reads OPENAI_API_KEY
embedder = SentenceTransformer("all-MiniLM-L6-v2")  # must match the index

def hyde_vector(question, n=3):
    passages = []
    for _ in range(n):
        response = client.responses.create(
            model="your-model-name",                    # any current chat model
            input=("Write a 2-3 sentence passage from an Indian college student "
                   f"handbook that answers this question:\n{question}"),
        )
        passages.append(response.output_text)
    vectors = embedder.encode([question] + passages, normalize_embeddings=True)
    v = 0.5 * vectors[0] + 0.5 * vectors[1:].mean(axis=0)   # keep the question anchored
    return v / np.linalg.norm(v)

query_vec = hyde_vector("Can I get my money back if I quit after two weeks?")
# results = collection.query(query_embeddings=[query_vec.tolist()], n_results=5)

Three practical notes on that code:

• Generate in parallel. Three sequential LLM calls triple the delay; send them concurrently, or ask for three passages in one call.
• Keep passages short. Two or three sentences in the documents' style is enough. Long passages drift off topic.
• Cache by question. Popular questions repeat, so store their HyDE vectors.

How HyDE compares with other query tricks

They combine well. A common stack is rewriting for chat context, then HyDE or multi-query for recall, then re-ranking for precision.

Common mistakes with HyDE

• Putting the hypothetical passage in the final prompt. It is invented text. Only real retrieved chunks are evidence.
• Using a different embedding model for the passage than for the index. Vectors must share one space.
• Dropping the original question from the search vector, which hurts exact-term queries like "ATKT".
• Never measuring. HyDE helps some collections and hurts others. Compare recall@5 with and without it on your own question set, as in evaluating RAG systems.

Key takeaways

• HyDE searches with an LLM-generated hypothetical answer instead of the question.
• It closes the gap between casual questions and formal documents.
• Wrong facts in the hypothetical are acceptable, because only real chunks reach the prompt.
• It can drift on jargon; average several passages with the question, and pair with BM25.
• It costs an extra LLM call, so measure recall before and after.

To try HyDE alongside rewriting and re-ranking on a real project, a classroom GenAI course that covers retrieval tricks like HyDE is available at Debugshala in Indore. You can also check your Python fundamentals with quick MCQs on Debugshala Test.