Embeddings Explained: How AI Turns Meaning into Vectors

Think about how you would describe dishes at Chappan Dukan to a friend using only scores out of 10. Jalebi: sweetness 9, spice 1, fried 9. Rasgulla: sweetness 9, spice 0, fried 0. Samosa: sweetness 1, spice 7, fried 9.

With just three numbers per dish you can already reason about them. Jalebi and rasgulla are close on sweetness. Jalebi and samosa are close on "fried". Something with sweetness 8, spice 1, fried 8 is probably a cousin of jalebi, even if you have never heard its name.

You just invented embeddings. Real ones work the same way, except a model chooses the scores and uses hundreds of dimensions instead of three.

What are embeddings?

Mapping the analogy: each dish is a piece of text, the three scores form its vector, each score is a dimension, and "dishes with similar scores taste alike" is the core promise: similar meaning, nearby vectors.

An embedding is a fixed-length vector of numbers, produced by a trained model, that places a piece of text in a space where distance and direction reflect meaning.

The big difference from your dish scores is that real dimensions are not labelled. No single number means "sweetness". The model learns, from huge amounts of text, an arrangement in which related texts end up close together. Typical embedding sizes range from a few hundred to a few thousand dimensions.

Embeddings power semantic search, recommendations, clustering, duplicate detection and the retrieval step of RAG systems. They are also how an LLM represents tokens internally, before any prediction happens.

Words and sentences as vectors

Related items cluster together. A small angle between two vectors means similar meaning; a wide angle means unrelated.

Early methods produced word embeddings: one vector per word, learned from which words appear near each other. The famous result was that vector arithmetic captured relationships, such as "king" minus "man" plus "woman" landing near "queen".

Word vectors have a limit: "bank" gets one vector whether you mean a river bank or SBI. Modern sentence embedding models read the whole text with a transformer, so the same word gets a different meaning from its context, and the output is one vector for the whole sentence or paragraph.

Cosine similarity: measuring how close two meanings are

The standard way to compare embeddings is cosine similarity: the cosine of the angle between two vectors. It is the dot product divided by the product of their lengths.

• 1.0: same direction, very similar meaning
• 0: at right angles, unrelated
• -1.0: opposite directions (rare in practice for text embeddings)

Cosine ignores vector length and looks only at direction. That is useful because a long document and a short one about the same topic should still count as similar.

Here is cosine similarity on the dish vectors from the opening, plus two vehicles, written with numpy. If arrays are new to you, the official numpy documentation is a good companion.

import numpy as np

# dimensions: [sweetness, spice, fried, has_wheels]
items = {
    "jalebi":   np.array([9, 1, 9, 0]),
    "rasgulla": np.array([9, 0, 0, 0]),
    "samosa":   np.array([1, 7, 9, 0]),
    "scooter":  np.array([0, 0, 0, 9]),
}

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

for name, vec in items.items():
    print(f"jalebi vs {name:<9}: {cosine(items['jalebi'], vec):.3f}")

Jalebi vs rasgulla scores about 0.71 and jalebi vs samosa about 0.66, because each pair shares one strong trait. Jalebi vs scooter scores 0, since they share nothing. Now see why direction, not size, matters:

import numpy as np

small = np.array([1.0, 2.0, 0.5])
big = small * 10            # same direction, ten times longer
other = np.array([2.0, -1.0, 0.0])

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

print("Euclidean distance small-big:", round(float(np.linalg.norm(small - big)), 2))
print("Cosine similarity small-big :", round(float(cosine(small, big)), 3))
print("Cosine similarity small-other:", round(float(cosine(small, other)), 3))

# Normalise to length 1 and cosine becomes a plain dot product
unit = lambda v: v / np.linalg.norm(v)
print("Dot of unit vectors:", round(float(unit(small) @ unit(big)), 3))

The last line is a practical trick. Many vector databases store normalised vectors so they can use the faster dot product and still get cosine similarity.

A bag-of-words embedding you can build yourself

Before learned models, a common way to turn sentences into vectors was bag of words: one dimension per vocabulary word, holding how many times it appears. Let us build one and see what it gets right and wrong.

import numpy as np

sentences = [
    "cheap hostel near vijay nagar",
    "affordable pg close to vijay nagar",
    "cheap flights to delhi",
]
vocab = sorted({w for s in sentences for w in s.split()})

def bow(sentence):
    words = sentence.split()
    return np.array([words.count(w) for w in vocab], dtype=float)

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

vecs = [bow(s) for s in sentences]
print("Vocabulary size:", len(vocab))
print("Hostel vs PG     :", round(float(cosine(vecs[0], vecs[1])), 3))
print("Hostel vs flights:", round(float(cosine(vecs[0], vecs[2])), 3))

The hostel and PG sentences mean nearly the same thing, yet bag of words scores them only 0.37, giving credit just for "vijay nagar". It also links "cheap hostel" to "cheap flights" just because both say "cheap". The method matches words, not meaning.

A trained embedding model has seen millions of sentences where "affordable" and "cheap", or "PG" and "hostel", appear in similar contexts. So it places them close together, and the hostel and PG sentences would score far higher than the flights one. That gap is the whole reason learned embeddings exist.

Semantic search with embeddings

Semantic search finds documents by meaning rather than exact keywords. The pipeline has two halves:

1. Index once: embed every document (or chunk of one) and store the vectors.
2. Query many times: embed the user's question with the same model, compute similarity against stored vectors, and return the top few.

Documents are embedded once; each query is embedded at search time and compared against them.

Here is the search step with hand-made "embeddings" for four help-desk articles. In a real system a model would produce these vectors.

import numpy as np

# pretend embeddings with dimensions: [payments, delivery, account, refunds]
docs = {
    "How to get your money back":    np.array([0.3, 0.0, 0.1, 0.9]),
    "Tracking a late parcel":        np.array([0.0, 0.9, 0.1, 0.1]),
    "Resetting your password":       np.array([0.0, 0.0, 0.95, 0.0]),
    "UPI payment failed but debited": np.array([0.9, 0.0, 0.1, 0.4]),
}
query = np.array([0.5, 0.0, 0.0, 0.8])   # "I want a refund for a failed payment"

names = list(docs)
matrix = np.array([docs[n] for n in names])
scores = matrix @ query / (np.linalg.norm(matrix, axis=1) * np.linalg.norm(query))

for i in np.argsort(-scores)[:3]:
    print(f"{scores[i]:.3f}  {names[i]}")

The query never says "money back", yet the refund article ranks first. That is semantic search. With millions of vectors, systems use approximate nearest neighbour indexes so they do not compare against every vector one by one.

Common mistakes with embeddings

• Mixing models. Vectors from two different embedding models live in different spaces. Embed queries and documents with the same model, and re-embed everything if you switch.
• Embedding huge documents whole. One vector for a 50-page PDF blurs every topic together. Split into chunks first.
• Trusting a fixed threshold. A score of 0.8 means different things on different models. Tune cut-offs on your own data.
• Ignoring keyword search. Exact codes, names and IDs ("Form 16", "IFSC") are often matched better by keyword search. Many production systems combine both, called hybrid search.
• Forgetting language coverage. Check that your embedding model handles Hindi or Hinglish if your users write that way.

Key takeaways

• Embeddings turn text into vectors where nearby means similar.
• Cosine similarity compares direction and ranges from -1 to 1.
• Bag of words matches words; learned embeddings match meaning.
• Semantic search embeds documents once and each query at search time, then ranks by similarity.
• Use one model for everything, chunk long text, and tune thresholds on real data.

Next, see how embeddings feed an LLM with the right facts in What is RAG?, and practise Python and aptitude questions on Debugshala Test to build fluency.