Embeddings and similarity search

An embedding is a list of numbers that stands for a piece of text, an image or anything else, arranged so that things with similar meaning get nearby lists. That one idea underlies semantic search, recommendation, clustering of documents, deduplication and the retrieval half of retrieval-augmented generation. This lesson is for anyone who has heard “we embed the documents and search the vectors” and wants to know what the numbers are, how “nearby” is measured, and what a search over them looks like in twenty lines of numpy. Everything runs on a laptop CPU with a model that is already cached.

From one-hot to dense

The networks in lesson 07 took pixel values as input. Text has no such natural numbers. The naive encoding gives every word in the vocabulary its own position in a very long vector, with a 1 in that position and 0 elsewhere. Those one-hot vectors are useless for similarity: “cat” and “kitten” are as far apart as “cat” and “carburettor”, because every pair of distinct words is orthogonal.

An embedding replaces the long sparse vector with a short dense one, typically a few hundred numbers, learned so that geometry reflects usage. Words or sentences that appear in similar contexts are pushed together during training, and ones that do not are pushed apart. The individual dimensions have no fixed meaning. What matters is the relative position of vectors.

Cosine similarity by hand

Imagine a toy embedding with three dimensions that happen to mean furriness, pet-ness and vehicle-ness. Three items get the vectors below. The question is how to turn two vectors into one similarity number.

import numpy as np

def cosine(u: np.ndarray, v: np.ndarray) -> float:
    return float(u @ v / (np.linalg.norm(u) * np.linalg.norm(v)))

cat = np.array([0.9, 0.8, 0.1])     # (furry, pet, vehicle)
dog = np.array([0.8, 0.9, 0.0])
car = np.array([0.0, 0.1, 0.9])

print(f"cat . dog = {cat @ dog:.2f}, |cat| = {np.linalg.norm(cat):.3f}, |dog| = {np.linalg.norm(dog):.3f}")
print(f"cos(cat, dog) = {cosine(cat, dog):.3f}")
print(f"cos(cat, car) = {cosine(cat, car):.3f}")
print(f"cos(cat, 3*cat) = {cosine(cat, 3 * cat):.3f}   # scale does not matter")

cat . dog = 1.44, |cat| = 1.208, |dog| = 1.204
cos(cat, dog) = 0.990
cos(cat, car) = 0.155
cos(cat, 3*cat) = 1.000   # scale does not matter

Follow the arithmetic once by hand. The dot product of cat and dog is 0.9 x 0.8 + 0.8 x 0.9 + 0.1 x 0.0 = 0.72 + 0.72 + 0 = 1.44. The length of cat is the square root of 0.81 + 0.64 + 0.01 = 1.46, which is 1.208. Dividing the dot product by both lengths gives 1.44 / (1.208 x 1.204) = 0.990. The result is the cosine of the angle between the two vectors: 1 when they point the same way, 0 when they are perpendicular, negative when they oppose.

Dividing by the lengths is the point of using cosine rather than the raw dot product. The last line shows why: tripling every number in cat does not change its direction, so its similarity to itself stays 1. In real embeddings, vector length tends to vary with things like sentence length and word frequency that have little to do with meaning, and cosine removes that. In practice most systems normalise every vector to length 1 once, at indexing time, after which the dot product and the cosine are the same number and search is a single matrix multiplication.

A real sentence encoder

The model used below is sentence-transformers/all-MiniLM-L6-v2, a 6-layer transformer encoder with 22 M parameters that maps a sentence to a 384-dimensional vector. It was released by the Sentence-Transformers project in 2021 under the Apache-2.0 licence with weights and training code public, so it is open source rather than only open weights. Its tokeniser accepts up to 512 tokens but the model was trained on inputs of 128 word pieces and truncates to 256 by default, so it is a model for sentences and short paragraphs, not for whole documents. It is small enough to encode thousands of sentences per second on a laptop CPU, which is why it appears in so many tutorials and why the Field Guide caches it for offline use.

How a sentence becomes a vector and how a query is matched against stored vectors. Everything to the left of the dotted line happens once per sentence at indexing time.

The tokeniser splits text into word pieces from a fixed vocabulary. The transformer turns each piece into a context-aware vector (the mechanism is the subject of the transformers hub). Mean pooling averages those piece vectors into one vector for the whole sentence, and normalisation scales it to unit length. The twenty sentences below were written for this lesson, in ten pairs on ten unrelated subjects, so that you can judge the search results yourself.

import numpy as np
from sentence_transformers import SentenceTransformer

sentences = [
    "The cat slept on the warm windowsill all afternoon.",
    "A kitten chased a ball of wool across the kitchen floor.",
    "Our dog barks whenever the postman arrives.",
    "The puppy dug a hole under the garden fence.",
    "Train services from Leeds were cancelled because of flooding.",
    "The 07:42 to Manchester is running twelve minutes late.",
    "Interest rates were held at 4 percent by the central bank.",
    "Inflation fell for the third month in a row.",
    "The recipe needs two eggs, flour and a pinch of salt.",
    "Whisk the batter until no lumps remain, then rest it for an hour.",
    "The goalkeeper saved a penalty in the final minute.",
    "The match ended two all after extra time.",
    "Jupiter is the largest planet in the solar system.",
    "The probe sent back images of Saturn's rings.",
    "Replace the brake pads when they are thinner than three millimetres.",
    "The mechanic changed the oil and rotated the tyres.",
    "She learned to play the cello at the age of nine.",
    "The orchestra tuned up before the symphony began.",
    "Back up your files before upgrading the operating system.",
    "The laptop battery lasts about six hours on a full charge.",
]

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cpu")
emb = model.encode(sentences, normalize_embeddings=True)   # unit length, so dot = cosine
print("embeddings:", emb.shape, emb.dtype)
print("first five values of sentence 0:", np.round(emb[0, :5], 3))
print("norm of sentence 0:", round(float(np.linalg.norm(emb[0])), 3))

embeddings: (20, 384) float32
first five values of sentence 0: [ 0.105  0.053  0.017  0.148 -0.028]
norm of sentence 0: 1.0

Twenty sentences became a 20 by 384 matrix of small floats. The individual values mean nothing you can name. The first time the model loads from the local cache takes a second or two, and encoding twenty sentences takes a few milliseconds after that. If the model is not cached, SentenceTransformer(...) downloads about 90 MB from the Hugging Face Hub, which the Hugging Face platform page explains how to manage.

Nearest-neighbour search in numpy

With unit-length vectors, the similarity of a query to every stored sentence is one matrix-vector product. Sorting the result gives the nearest neighbours.

import numpy as np
from sentence_transformers import SentenceTransformer

sentences = [
    "The cat slept on the warm windowsill all afternoon.",
    "A kitten chased a ball of wool across the kitchen floor.",
    "Our dog barks whenever the postman arrives.",
    "The puppy dug a hole under the garden fence.",
    "Train services from Leeds were cancelled because of flooding.",
    "The 07:42 to Manchester is running twelve minutes late.",
    "Interest rates were held at 4 percent by the central bank.",
    "Inflation fell for the third month in a row.",
    "The recipe needs two eggs, flour and a pinch of salt.",
    "Whisk the batter until no lumps remain, then rest it for an hour.",
    "The goalkeeper saved a penalty in the final minute.",
    "The match ended two all after extra time.",
    "Jupiter is the largest planet in the solar system.",
    "The probe sent back images of Saturn's rings.",
    "Replace the brake pads when they are thinner than three millimetres.",
    "The mechanic changed the oil and rotated the tyres.",
    "She learned to play the cello at the age of nine.",
    "The orchestra tuned up before the symphony began.",
    "Back up your files before upgrading the operating system.",
    "The laptop battery lasts about six hours on a full charge.",
]
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cpu")
emb = model.encode(sentences, normalize_embeddings=True)

def search(query: str, k: int = 3) -> None:
    q = model.encode([query], normalize_embeddings=True)[0]
    scores = emb @ q                          # one cosine per stored sentence
    top = np.argsort(-scores)[:k]
    print(f"\nquery: {query}")
    for i in top:
        print(f"  {scores[i]:.3f}  {sentences[i]}")

search("my pet is asleep in the sun")
search("delayed railway journey")
search("how do I make pancakes")

query: my pet is asleep in the sun
  0.490  The cat slept on the warm windowsill all afternoon.
  0.355  Our dog barks whenever the postman arrives.
  0.171  The puppy dug a hole under the garden fence.

query: delayed railway journey
  0.363  The 07:42 to Manchester is running twelve minutes late.
  0.359  Train services from Leeds were cancelled because of flooding.
  0.160  Whisk the batter until no lumps remain, then rest it for an hour.

query: how do I make pancakes
  0.523  The recipe needs two eggs, flour and a pinch of salt.
  0.325  Whisk the batter until no lumps remain, then rest it for an hour.
  0.104  Replace the brake pads when they are thinner than three millimetres.

None of the three queries shares a content word with its best match. “Pet” and “asleep” found “cat” and “slept”. “Railway” found “train” and “the 07:42 to Manchester”. “Pancakes” found eggs, flour and batter. That is the capability the model was trained for, and the scores tell you how confident to be: the top hits are around 0.35 to 0.52, the third result in each case is around 0.1 to 0.17 and is a different subject. A threshold near 0.3 would separate hits from misses on this small set, though the right threshold depends on the model and the collection and has to be checked on your own data.

The absolute values are lower than the toy example because real embeddings spread meaning across 384 dimensions and two sentences about the same subject still differ in most of them. Compare scores within a query, not across models.

np.argsort over twenty rows is instant. Over a million rows a single query is still only a 1 million by 384 matrix-vector product, about 0.1 s on a laptop, and over tens of millions you move to an approximate nearest-neighbour index that trades a little recall for speed. The RAG pipeline checklist covers that step, and the tiny RAG project builds a complete retrieval pipeline on exactly this model.

The same search with an inverted index

Before embeddings, search meant an inverted index: for every word, the list of documents containing it. The block below builds one over the same twenty sentences, with a handful of stop words removed so that “the” does not match everything, and runs the same three queries plus one more.

import re
from collections import defaultdict

sentences = [
    "The cat slept on the warm windowsill all afternoon.",
    "A kitten chased a ball of wool across the kitchen floor.",
    "Our dog barks whenever the postman arrives.",
    "The puppy dug a hole under the garden fence.",
    "Train services from Leeds were cancelled because of flooding.",
    "The 07:42 to Manchester is running twelve minutes late.",
    "Interest rates were held at 4 percent by the central bank.",
    "Inflation fell for the third month in a row.",
    "The recipe needs two eggs, flour and a pinch of salt.",
    "Whisk the batter until no lumps remain, then rest it for an hour.",
    "The goalkeeper saved a penalty in the final minute.",
    "The match ended two all after extra time.",
    "Jupiter is the largest planet in the solar system.",
    "The probe sent back images of Saturn's rings.",
    "Replace the brake pads when they are thinner than three millimetres.",
    "The mechanic changed the oil and rotated the tyres.",
    "She learned to play the cello at the age of nine.",
    "The orchestra tuned up before the symphony began.",
    "Back up your files before upgrading the operating system.",
    "The laptop battery lasts about six hours on a full charge.",
]

STOP = {"the", "a", "an", "is", "in", "on", "of", "to", "and", "my", "our", "i", "do", "how"}

def tokens(text: str) -> set[str]:
    return set(re.findall(r"[a-z0-9]+", text.lower())) - STOP

index: dict[str, set[int]] = defaultdict(set)
for i, s in enumerate(sentences):
    for t in tokens(s):
        index[t].add(i)
print("distinct terms:", len(index), " postings for 'train':", sorted(index["train"]))

def keyword_search(query: str, k: int = 3) -> None:
    hits: dict[int, int] = defaultdict(int)
    for t in tokens(query):
        for i in index.get(t, ()):
            hits[i] += 1
    ranked = sorted(hits.items(), key=lambda kv: -kv[1])[:k]
    print(f"\nquery: {query}")
    if not ranked:
        print("  (no sentence shares a word with the query)")
    for i, n in ranked:
        print(f"  {n} shared term(s)  {sentences[i]}")

keyword_search("my pet is asleep in the sun")
keyword_search("delayed railway journey")
keyword_search("how do I make pancakes")
keyword_search("worn brake pads")

distinct terms: 132  postings for 'train': [4]

query: my pet is asleep in the sun
  (no sentence shares a word with the query)

query: delayed railway journey
  (no sentence shares a word with the query)

query: how do I make pancakes
  (no sentence shares a word with the query)

query: worn brake pads
  2 shared term(s)  Replace the brake pads when they are thinner than three millimetres.

The three paraphrased queries return nothing, because not one of their words appears in the collection. This is the vocabulary mismatch problem and it is the reason embeddings took over for natural-language queries. The fourth query shows the other side. “Brake pads” is an exact term and the index finds it with certainty, in constant time, and would do the same for a part number, an error code or a person’s name. Embedding models are weaker on exactly those: rare tokens get split into word pieces the model has little signal for, and two different part numbers can land close together.

	Inverted index (BM25 and relatives)	Embedding search
Finds paraphrases	No	Yes
Exact identifiers, codes, names	Reliable	Unreliable
Index cost	Tiny, no model	One forward pass per document
Explains a match	Yes, the shared terms	No
Needs tuning for a new domain	Rarely	Sometimes, the model may need fine-tuning

Production systems usually run both and merge the two ranked lists, which is called hybrid search. The retrieval hub covers the merging and the reranking step that often follows.

Exercise

Add five sentences of your own to the list, on a subject not already covered, and write a query that should find them without sharing a word. Report the top-3 scores and whether a 0.3 threshold would have worked.
Replace normalize_embeddings=True with False in the search block and change scores = emb @ q to compute cosine explicitly with the norms. Confirm the rankings are the same, then print the norms of the twenty raw vectors and see how much they vary.
Extend the keyword index to score by the number of shared terms divided by the number of terms in the sentence, so that short exact matches rank above long partial ones. Run “worn brake pads” and “oil change” and compare with the embedding results for the same two queries.

Summary

An embedding is a dense vector learned so that similar inputs land close together. The dimensions have no individual meaning, only relative position does.
Cosine similarity is the dot product divided by both lengths. It ignores vector scale, and after normalising vectors to unit length it is a plain dot product.
all-MiniLM-L6-v2 (22 M parameters, 384 dimensions, Apache-2.0) encodes a sentence in milliseconds on CPU and finds paraphrases that share no words with the query.
Nearest-neighbour search over normalised vectors is one matrix-vector product and an argsort. Approximate indexes are needed only at millions of rows.
Keyword indexes fail on paraphrase but win on exact identifiers and explain their matches. Most systems use both.