# Attention

Course: [Large Language Models](https://www.devtutorials.ai/learn/llm) · Section: Inside LLMs
Source: https://www.devtutorials.ai/learn/llm/inside-llms/attention

> How attention lets each token look back at earlier tokens and decide which ones matter, with scores, softmax weights, mixing, the causal mask and attention heads.

Read this sentence: "The animal did not cross the street because **it** was tired."

What does "it" mean? You know right away that "it" refers to the animal, not the street. Streets do not get tired. To figure that out, you looked back at the other words and picked the one that mattered.

An LLM has to do the same thing for every token. The part of the model that does this job is called **attention**.

**The main idea:** attention lets each token look back at the earlier tokens, decide which ones matter most, and mix in their meaning.

## A Real-World Example

Imagine a student in a classroom who has just been asked a question.

The student looks around the room. Some classmates know a lot about this topic. Others do not. The student listens closely to the helpful ones, listens a little to the others, and then puts together an answer from what they heard.

Attention works in the same way.

- **The student** is the token that is being updated, like "it".
- **The classmates** are the earlier tokens in the text.
- **How closely the student listens** is the attention weight for each token.
- **The final answer** is a new embedding for "it", mixed together from the helpful tokens.

**In simple words:** each token asks the earlier tokens "who can help me?" and listens most to the best answers.

## What You Will Learn

In this lesson, you will learn:

1. What problem attention solves.
2. How a token gives each earlier token a score.
3. How softmax turns scores into weights that add up to 100%.
4. How the weights mix information into a new embedding.
5. Why a token can only look back, never ahead.
6. What attention heads and layers are.

These code boxes use tiny, made-up numbers so you can follow each step. A real model does exactly these steps, but with vectors of thousands of numbers and many heads at once.

## The Problem Attention Solves

In the Embeddings lesson, each token got one fixed embedding from the table. That embedding is the same in every sentence.

This is a problem. Many words change their meaning based on the words around them.

- "bank" in "river bank" and "bank account"
- "it" in any sentence, which can point to almost anything
- "cold" in "a cold drink" and "a cold person"

Attention fixes this. It updates each token's embedding using the tokens around it. After attention, "it" carries some of the meaning of "animal".

## Step 1: Scores

First, the token gives every earlier token a **score**. A high score means "you look useful to me".

To do this, every token makes two small vectors from its embedding:

- a **query**, which means "what am I looking for?"
- a **key**, which means "what do I have to offer?"

The score is the **dot product** of one token's query and another token's key: multiply the matching numbers and add them up. When the query and the key point the same way, the score is high.

```js runnable title="Score each earlier word for the word 'it'"
const query = [1.0, 0.5];  // what "it" is looking for

const keys = {
  animal: [1.2, 0.8],
  cross:  [0.2, 0.3],
  street: [0.6, 0.1],
  tired:  [0.4, 1.0],
};

for (const word in keys) {
  const key = keys[word];
  const score = query[0] * key[0] + query[1] * key[1];
  console.log(word.padEnd(6), score.toFixed(2));
}
```

Expected output:

```text
animal 1.60
cross  0.35
street 0.65
tired  0.90
```

`animal` gets the highest score. In a real model, the query and key vectors are learned during training, so useful matches get high scores.

## Step 2: Softmax Turns Scores Into Weights

Scores can be any size. To use them, the model turns them into **weights** that are all positive and add up to 100%. The function that does this is called **softmax**.

Softmax does two things:

1. It makes every score positive with the exponent function, `Math.exp()`.
2. It divides each one by the total, so they add up to 1, which is 100%.

```js runnable title="Softmax: scores to weights"
function softmax(scores) {
  const exps = scores.map((s) => Math.exp(s));
  const total = exps.reduce((a, b) => a + b, 0);
  return exps.map((e) => e / total);
}

const words = ["animal", "cross", "street", "tired"];
const scores = [1.6, 0.35, 0.65, 0.9];
const weights = softmax(scores);

words.forEach((word, i) => {
  console.log(word.padEnd(6), (weights[i] * 100).toFixed(0) + "%");
});
console.log("Total:", Math.round(weights.reduce((a, b) => a + b, 0) * 100) + "%");
```

Expected output:

```text
animal 46%
cross  13%
street 18%
tired  23%
Total: 100%
```

"it" will now pay 46% of its attention to "animal" and much less to the other words. You will meet softmax again in the Logits lesson. It is the same function that turns a model's scores into next-token chances.

**In simple words:** softmax turns "how useful?" scores into "how much to listen" percentages.

## Step 3: Mix the Information

Each token also makes a third vector, a **value**. The value is "the information I will share".

The new embedding for "it" is a **weighted average** of the values. Tokens with bigger weights add more of their information.

```js runnable title="Mix the values with the weights"
const values = {
  animal: [1, 0, 0],
  cross:  [0, 1, 0],
  street: [0, 0, 1],
  tired:  [0, 1, 1],
};
const weights = { animal: 0.46, cross: 0.13, street: 0.18, tired: 0.23 };

const mixed = [0, 0, 0];
for (const word in values) {
  for (let i = 0; i < 3; i++) {
    mixed[i] += weights[word] * values[word][i];
  }
}

console.log("New vector for 'it':", mixed.map((x) => x.toFixed(2)).join(", "));
```

Expected output:

```text
New vector for 'it': 0.46, 0.36, 0.41
```

The first number, which stands for "animal" in this toy example, is now the biggest part of the new vector for "it". The token "it" now carries the meaning of "animal".

**Easy Way to Remember:** attention is **score, softmax, mix**. Score every earlier token, turn the scores into percentages, and mix in their information.

## Looking Back Only

When an LLM writes text, it makes one token at a time. The future tokens do not exist yet. So during training and generation, each token is only allowed to look at itself and the tokens **before** it.

This rule is called the **causal mask**. "Causal" means the past can affect the future, but the future cannot affect the past.

```js runnable title="What each token is allowed to see"
const words = ["The", "cat", "sat", "down"];

for (let i = 0; i < words.length; i++) {
  const visible = words.slice(0, i + 1);
  console.log(words[i].padEnd(4), "can see:", visible.join(" "));
}
```

Expected output:

```text
The  can see: The
cat  can see: The cat
sat  can see: The cat sat
down can see: The cat sat down
```

The first token can only see itself. The last token can see everything. This is why the model can guess the next token: the last token has gathered information from the whole text so far.

## Heads and Layers

A real model does not run attention just once.

### Attention Heads

Each attention step has many **heads** running side by side. Each head has its own queries and keys, so each one can look for something different.

- One head might find which noun a word like "it" points to.
- Another head might link a word to the word just before it.
- Another might match an opening bracket with its closing bracket in code.

Nobody tells the heads what to look for. These jobs appear during training.

```js runnable title="Two heads look for different things"
function softmax(scores) {
  const exps = scores.map((s) => Math.exp(s));
  const total = exps.reduce((a, b) => a + b, 0);
  return exps.map((e) => e / total);
}

const words = ["animal", "street", "tired"];
const heads = {
  "Head 1 (who?)": [3.0, 0.5, 0.2],
  "Head 2 (how?)": [0.3, 0.2, 2.5],
};

for (const head in heads) {
  const weights = softmax(heads[head]);
  let best = 0;
  weights.forEach((w, i) => { if (w > weights[best]) best = i; });
  console.log(head + ": looks most at '" + words[best] + "' (" + (weights[best] * 100).toFixed(0) + "%)");
}
```

Expected output:

```text
Head 1 (who?): looks most at 'animal' (87%)
Head 2 (how?): looks most at 'tired' (83%)
```

Head 1 answers "who is it?". Head 2 answers "how is it?". The results of all heads are joined together, so "it" learns both.

### Layers

Attention is one part of a **layer**. A layer is attention plus a small network that processes each token on its own. A model stacks many layers, often 24 to 100 or more.

- Early layers tend to pick up simple patterns, like nearby words.
- Later layers build bigger ideas, like who did what in a story.
- After the last layer, the final token's vector is used to guess the next token.

This whole design, built from stacked attention layers, is called a **transformer**. Almost every modern LLM is a transformer.

Attention is not the same as "understanding". It is a way to mix information using learned scores. It works very well, but it can still link the wrong words, especially in long or confusing sentences.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Fixed meanings | "Each token always has the same meaning." | Attention updates each token's vector using its context. | You cannot explain how "bank" means two things. |
| Weights from nowhere | "Someone sets which words to look at." | The queries and keys are learned, so the scores are learned too. | You expect to edit attention by hand. |
| Looking ahead | "A token can use words that come later." | The causal mask lets a token see only itself and earlier tokens. | You expect the model to plan around words it has not written yet. |
| Forgetting softmax | "The raw scores are the weights." | Softmax turns scores into positive weights that add up to 100%. | Mixing with raw scores gives wrong, unbalanced results. |
| One head is enough | "Attention looks for one thing at a time." | Many heads run together, each finding a different pattern. | You underestimate how much context the model can use. |
| Attention means understanding | "If it attends to a word, it understands it." | Attention mixes information. It can still link the wrong words. | You trust answers about confusing or very long text too much. |

## Exercises

Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer.

## What to Remember

- **Attention** lets each token look at earlier tokens and mix in the useful ones.
- Each token makes a **query**, a **key**, and a **value** from its embedding.
- A **score** is the dot product of a query and a key.
- **Softmax** turns scores into weights that add up to 100%.
- The new vector is a **weighted mix** of the values.
- The **causal mask** means a token only looks back, never ahead.
- Many **heads** look for different things. Many **layers** are stacked to make a **transformer**.

Next: **Model Training**, how all these numbers are learned in the first place.