# Embeddings

Course: [Large Language Models](https://www.devtutorials.ai/learn/llm) · Section: Inside LLMs
Source: https://www.devtutorials.ai/learn/llm/inside-llms/embeddings

> How a model turns each token into a list of numbers that carries meaning, how similarity between vectors is measured, and how real embeddings group similar text.

A token ID is just a number, such as 11 or 4,213. The number by itself has no meaning. Token 11 is not "smaller" than token 4,213 in any useful sense.

So the first thing a model does with each token ID is trade it for something much richer: a long list of numbers that describes what the token means. This list of numbers is called an **embedding**.

**The main idea:** an embedding turns a token into a point in space, and tokens with similar meanings end up close to each other.

## A Real-World Example

Imagine a map of a city.

Every shop on the map has an address made of two numbers, like a latitude and a longitude. Shops of the same kind are often found near each other. All the cafés are on one street, close together. All the car garages sit together near the highway.

If you know two addresses, you can tell how close two shops are without ever going there.

Embeddings work the same way, like map addresses for words.

- **A shop** is a token or a small piece of text.
- **Its address** is the embedding, which is a list of numbers.
- **Shops of the same kind that are nearby** stand for words with similar meanings that are close together.
- **Measuring the distance** between addresses is how a computer compares the meanings of words.

A city map uses 2 numbers for each address. Real embeddings use hundreds or even thousands of numbers. The idea is still the same.

**In simple words:** an embedding is an address for meaning.

## What You Will Learn

In this lesson, you will learn:

1. What an embedding is, and where it comes from.
2. Why similar words get similar numbers.
3. How to measure distance between two embeddings.
4. How cosine similarity compares directions.
5. How to find the nearest neighbours of a word.
6. How real sentence embeddings group text by meaning.

The code boxes use tiny, made-up embeddings with 2 or 3 numbers, so you can read every value. The Embedding Lab further down uses a real embedding model with 384 numbers per sentence.

## What Is an Embedding?

An **embedding** is a list of numbers that stands for a token or a piece of text. A list of numbers like this is also called a **vector**.

- Each number describes one hidden feature of the meaning.
- No person picks these features by hand. The model learns them by itself during training.
- Similar meanings end up with similar lists of numbers.

Below are four words with tiny two-number embeddings. You can think of the numbers as an x and a y position on a map.

```js runnable title="Words as lists of numbers"
const words = {
  cat:   [0.9, 0.8],
  dog:   [0.85, 0.75],
  car:   [0.1, 0.9],
  truck: [0.15, 0.95],
};

for (const word in words) {
  console.log(word.padEnd(6), words[word].join(", "));
}
```

Expected output:

```text
cat    0.9, 0.8
dog    0.85, 0.75
car    0.1, 0.9
truck  0.15, 0.95
```

`cat` and `dog` have very similar numbers. `car` and `truck` also have very similar numbers. The animals and the vehicles are far apart from each other.

## Where Embeddings Live in a Model

Inside the model, there is a large table called the **embedding table**. It has one row for every token in the vocabulary.

1. The tokenizer turns text into token IDs.
2. Each token ID picks its row from the embedding table.
3. That row, a list of numbers, is what the rest of the model works with.

```js runnable title="Look up token IDs in the embedding table"
const embeddingTable = [
  [0.0, 0.0, 0.0],   // row 0
  [0.9, 0.8, 0.1],   // row 1: "cat"
  [0.85, 0.75, 0.2], // row 2: "dog"
  [0.1, 0.2, 0.95],  // row 3: "car"
];

const tokenIds = [1, 3];
for (const id of tokenIds) {
  console.log("Token", id, "->", embeddingTable[id].join(", "));
}
```

Expected output:

```text
Token 1 -> 0.9, 0.8, 0.1
Token 3 -> 0.1, 0.2, 0.95
```

A real model has more than 100,000 rows, and each row holds hundreds or even thousands of numbers. The numbers start out random. Training slowly moves them until similar tokens end up close together.

**In simple words:** the token ID is the row number, and the row itself is the embedding.

## Measuring Closeness

To compare two embeddings, you need a way to measure how close they are. There are two common ways to do this.

### Distance

The first way is the straight-line distance between two points, the way you would measure with a ruler on a map.

```js runnable title="Straight-line distance"
const cat = [0.9, 0.8];
const dog = [0.85, 0.75];
const car = [0.1, 0.9];

function distance(a, b) {
  let sum = 0;
  for (let i = 0; i < a.length; i++) {
    sum += (a[i] - b[i]) ** 2;
  }
  return Math.sqrt(sum);
}

console.log("cat to dog:", distance(cat, dog).toFixed(2));
console.log("cat to car:", distance(cat, car).toFixed(2));
```

Expected output:

```text
cat to dog: 0.07
cat to car: 0.81
```

A small distance means the two meanings are close. `cat` is much closer to `dog` than to `car`.

### Cosine Similarity

The second way is used most often with embeddings. It is called **cosine similarity**. It checks whether two vectors point in the **same direction**, and it does not care how long they are.

- **1** means they point the same way: very similar meaning.
- **0** means they are unrelated.
- **Below 0** means they point in opposite directions.

```js runnable title="Cosine similarity"
function cosine(a, b) {
  let dot = 0, lengthA = 0, lengthB = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    lengthA += a[i] * a[i];
    lengthB += b[i] * b[i];
  }
  return dot / (Math.sqrt(lengthA) * Math.sqrt(lengthB));
}

const cat = [0.9, 0.7, 0.1];
const dog = [0.7, 0.8, 0.3];
const car = [0.1, 0.2, 0.95];

console.log("cat vs dog:", cosine(cat, dog).toFixed(2));
console.log("cat vs car:", cosine(cat, car).toFixed(2));
```

Expected output:

```text
cat vs dog: 0.97
cat vs car: 0.29
```

`cat` and `dog` score 0.97, which is close to 1. `cat` and `car` score only 0.29.

**In simple words:** cosine similarity asks "do these two point the same way?"

### Nearest Neighbours

With cosine similarity, you can find the words closest to any given word. These closest words are called its **nearest neighbours**.

```js runnable title="Find the nearest neighbours"
function cosine(a, b) {
  let dot = 0, lengthA = 0, lengthB = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    lengthA += a[i] * a[i];
    lengthB += b[i] * b[i];
  }
  return dot / (Math.sqrt(lengthA) * Math.sqrt(lengthB));
}

const vectors = {
  cat:    [0.9, 0.7, 0.1],
  dog:    [0.7, 0.8, 0.3],
  kitten: [0.95, 0.5, 0.1],
  car:    [0.1, 0.2, 0.95],
  bus:    [0.2, 0.1, 0.9],
};

const query = "kitten";
const scores = [];
for (const word in vectors) {
  if (word !== query) scores.push([word, cosine(vectors[query], vectors[word])]);
}
scores.sort((a, b) => b[1] - a[1]);

for (const [word, score] of scores) {
  console.log(word.padEnd(4), score.toFixed(2));
}
```

Expected output:

```text
cat  0.98
dog  0.92
bus  0.33
car  0.28
```

`kitten` is closest to `cat`, and then to `dog`. The vehicles are far behind. This is the idea behind **semantic search**: finding text by meaning, not by matching the same words.

**Easy Way to Remember:** an embedding is an **address for meaning**. Close addresses mean close meanings.

## Directions Can Carry Meaning

Embeddings can do something surprising. Some **directions** in the space can carry meaning too. A well-known example: moving from `man` to `woman` is a similar step to moving from `king` to `queen`.

```js runnable title="king - man + woman"
const v = {
  king:  [0.9, 0.9, 0.1],
  man:   [0.5, 0.9, 0.1],
  woman: [0.5, 0.1, 0.1],
  queen: [0.9, 0.1, 0.1],
  apple: [0.1, 0.5, 0.9],
};

const result = v.king.map((x, i) => x - v.man[i] + v.woman[i]);
console.log("king - man + woman =", result.map((x) => x.toFixed(1)).join(", "));

let best = "";
let bestDistance = Infinity;
for (const word in v) {
  if (["king", "man", "woman"].includes(word)) continue;
  const d = Math.hypot(...v[word].map((x, i) => x - result[i]));
  if (d < bestDistance) {
    bestDistance = d;
    best = word;
  }
}
console.log("Closest word:", best);
```

Expected output:

```text
king - man + woman = 0.9, 0.1, 0.1
Closest word: queen
```

These numbers were made up to show the idea clearly. Real embeddings also show this pattern, but not as neatly.

## Try Real Embeddings

The lab below uses a real, small embedding model. It turns each sentence into a list of 384 numbers, and then compares each sentence with your question using cosine similarity.

Things to try:

- Look at the top result. Does it share words with your question, or only meaning?
- Change the question to "how much does it cost?" and compare again.
- Add a sentence in another language with the same meaning as your question.

This model makes **one embedding for a whole sentence**. That is different from the per-token embeddings inside an LLM. Both work on the same idea: similar meaning, similar numbers. Sentence embeddings like these power search and the RAG course later on.

A high similarity score means "close in meaning", not "correct" or "the answer". Two sentences can be close because they talk about the same topic while saying opposite things, like "the bug is fixed" and "the bug is not fixed".

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Treating IDs as meaning | "Token 12 is close to token 13, so they mean similar things." | IDs are just row numbers. The embedding row carries the meaning. | Comparisons based on IDs give nonsense. |
| Mixing models | "I can compare embeddings from any two models." | Each model has its own space. Only compare vectors from the same model. | Similarity scores look random. |
| Score means correct | "The top match must be the right answer." | A high score means similar topic or meaning, not correct facts. | Search returns a related but wrong document. |
| Reading single numbers | "Number 5 in the vector means 'animal'." | Meaning is spread across many numbers together. | You draw wrong conclusions from one value. |
| Using plain word matching | "Search for the exact words the user typed." | Embeddings match meaning, even with different words. | Good results are missed when the wording differs. |
| Confusing token and sentence embeddings | "The LLM makes one vector per sentence." | An LLM has one vector per token. Sentence models make one per text. | You use the wrong tool for search. |

## Exercises

Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer.

## What to Remember

- An **embedding** is a list of numbers, a **vector**, that stands for meaning.
- Similar meanings get **similar vectors** that sit close together.
- Inside an LLM, each token ID picks a row from the **embedding table**.
- **Cosine similarity** compares directions: near **1** means close in meaning.
- **Nearest neighbours** are the vectors closest to a given one.
- Sentence embedding models make **one vector per text**, which powers search.
- Only compare embeddings that come from the **same model**.

Next: **Attention**, how each token looks at the other tokens to understand its context.