The dot product (The Math Behind AI)

Multiply pair by pair, then add

Two lists in, one number out.

The same recipe in symbols: $a_1$ is the first number in list $\mathbf a$ , $a_2$ the second, and $a_i$ any one of them ( $i$ is just its position). $n$ is how many numbers each list holds. The big $\sum$ means “add them all up”, for $i$ = 1, 2, and so on up to $n$ (see how to read the math).

\mathbf a\cdot\mathbf b \;=\; \underbrace{a_1b_1+a_2b_2+\dots+a_nb_n}_{\ink{\text{multiply pair by pair, then add}}} \;=\; \sum_{i=1}^{n}a_i\,b_i

Now do it with your own numbers. The lab starts on the worked example above.

Multiply pair by pair, then addHands-on

Change any number with − and +. Every product and the sum update as you go. The bottom row (length, cosine and angle) comes up in the next section.

2 × 12

1 × 44

3 × 00

a·b =2 + 4 + 0 =6

length of a√14 = 3.742

length of b√17 = 4.123

cosine of the angle0.389

angle67.1°

Challenge: make the dot product exactly 0 without using a single 0.

What the number means

Draw each list as an arrow. Arrows that point the same way give a big positive number. Arrows at right angles give exactly zero, however long they are. Opposite arrows give a negative number. Drag the arrows below and watch the sign change.

What the number meansHands-on

Drag the tip of either arrow.

a = (3, 1) b = (1, 2)

a·b = (3)(1) + (1)(2) =5

they lean the same way

length of a3.162

length of b2.236

angle between them45.0°

cosine of that angle0.707

length × length × cosine5.000

The angle is measured from the drawing itself. The last box uses a second formula, shown just below the lab. Whenever neither arrow has length 0, it lands on the same number as the dot product.

Here’s that second formula:

\mathbf a\cdot\mathbf b \;=\; \underbrace{|\mathbf a|}_{\ink{\text{length of a}}}\;\underbrace{|\mathbf b|}_{\ink{\text{length of b}}}\;\underbrace{\cos\theta}_{\ink{\text{cosine of the angle }\theta\text{ between them}}}

An arrow’s length: square each of its numbers, add them up, and take the square root. That’s Pythagoras, and it works with any number of numbers.

$\theta$ , the Greek letter theta, is just a name for the angle. Its cosine is a number your calculator’s cos key gives you: 1 when the arrows point the same way, 0 at a right angle, and −1 when they point opposite ways.

Cosine similarity

The formula above says $\mathbf a\cdot\mathbf b=|\mathbf a|\,|\mathbf b|\cos\theta$ . Divide both sides by the two lengths and only $\cos\theta$ is left. It depends on the angle alone, not on how long the arrows are:

\cos\theta=\dfrac{\mathbf a\cdot\mathbf b}{|\mathbf a|\,|\mathbf b|}

That’s the score from the last lesson: Paris and France 0.47, Paris and banana 0.13. One dot product, two lengths, one division.

For the experts (skip it on a first read)

The two formulas agree in any number of dimensions. In high dimensions the first one is usually taken as the definition, and the second as the definition of the angle.

How the model scores Paris

One dot product of 1,024 pairs. Real numbers.

Skip to the very end of the model for a moment. After 28 layers of work, the last token, ·of, has turned into a new vector of 1,024 numbers that has soaked up the whole sentence. I’ll call it $\mathbf h$ , for “hidden”, because it’s a step inside the model, not its answer.

To score a candidate next word, the model takes one dot product: $\mathbf h$ with that word’s vector, the same row of the token table it would use to read that word in. (Not every model reuses its table this way; this small Qwen3 does.) Here are the first eight pairs for ·Paris:

Pair	h (the model’s summary of the sentence)	Paris’s vector	Product
1	−2.339	−0.0086060	0.020
2	7.124	−0.0024872	−0.018
3	−0.932	−0.031738	0.030
4	18.573	0.068848	1.279
5	1.605	−0.012146	−0.019
6	−14.467	−0.039795	0.576
7	5.828	0.034180	0.199
8	−60.550	−0.0028839	0.175
… 1,016 more pairs …			…
Sum of all 1,024 products: Paris’s score			22.08

The first eight products add up to 2.24. All 1,024 of them add up to 22.08. That’s Paris’s score; experts call it a logit. On its own a score means little: what counts is how it compares with every other word’s score. Each of the 151,936 rows in the table gets its own score the same way, and idea 3 (softmax) turns those scores into probabilities.

Notice how small that cosine is. Paris, the word that ends up winning, points only a little bit the same way as $\mathbf h$ . That’s plenty. Two arrows pointing in random directions, with 1,024 numbers each, almost always sit close to 90° apart: their 1,024 products come out plus and minus about equally, so they mostly cancel. Next to that, 79° is a real lean. By that yardstick even banana’s 0.13 with Paris (about 83°) is a small lean; in the last lesson, what mattered was that places lean much harder.

Where dot products hide

Every neuron (next lesson).
Every row of every matrix, the big grids of weights inside the model (idea 2): about 596 million weight multiplications per token, all inside dot products.
Every attention score (idea 3): a dot product of two vectors called a query and a key, which helps decide how much one token looks at another.
Every word score (this lesson).

For the experts (skip it on a first read)

$\mathbf h$ here is the last hidden state after the final RMS norm. Qwen3-0.6B-Base ties its output layer to its input table, so Paris’s “word vector” is the same row it used to read the word in. Two random directions in 1,024 dimensions are almost always close to 90° apart, which is why a cosine of 0.2 is a strong lean.

In the table, Paris’s numbers are shown to five significant digits and h to three decimals, so every row multiplies out to the product shown.