Before the five ideas
Words become numbers
A language model (the kind of AI you chat with) can’t read. Before any math can happen, a sentence has to turn into numbers: first into pieces called tokens, then into a long list of numbers for each piece. Everything the model does after this step is arithmetic on those lists.
In this lesson
- How a real model chops one sentence into 10 tokens
- What a 1,024-number vector is, and why it’s an arrow
- Why Paris points closer to France than to banana
The sentence we’ll follow
From here on, the course follows one sentence through a real AI model: “The Eiffel Tower is in the city of”. The model’s job is to guess the next word.
The model is Qwen3-0.6B-Base, a small open model made by Alibaba’s Qwen team in 2025. I rewrote its math in plain numpy, a basic math toolkit for the Python language. That way I can stop it at any step and print the numbers. I checked my version against the standard code from Hugging Face, the site that hosts the model.
Its top guess for the next word is Paris: it gives Paris a 63.3% chance. By the end of the course you’ll be able to follow that guess from start to finish.
Step 1: chop the text into tokens
Our sentence, exactly as Qwen3 sees it.
Qwen3 never sees letters. It sees tokens: pieces of text from a fixed list. A program called the tokenizer chose that list before the model was trained. Common words get a token of their own, usually with the space in front glued on (the little dot marks that space). Rarer words are built from smaller pieces. “Eiffel” isn’t common enough to earn its own token, so it becomes ·E iff el .
So our 8 words become 10 tokens, and each token is really just a number: its row in a big table inside the model, one row per token. ·Tower is row 21,938. From this point until it writes its answer, the model never touches text again.
The model we’re using
| Fact | Qwen3-0.6B-Base |
|---|---|
| Made by | Alibaba’s Qwen team, 2025 |
| Weights (learned numbers) | 596,049,920 |
| Layers (stages the numbers pass through, one after another) | 28 |
| Numbers per token | 1,024 |
| Rows in the token table | 151,936 |
| Trained on (its first, general training) | 36 trillion tokens, 119 languages |
Why such a small model? Because it runs on my laptop in plain numpy, where I can stop it at any step and print the numbers. Its giant cousins use the same five ideas, with more and bigger layers and a few extra tricks.
For the experts (skip it on a first read)
Qwen’s tokenizer is byte-level byte-pair encoding: it starts from raw bytes and glues neighbors into bigger pieces by a list of merges learned from the most frequent pairs in its training text, so any text in any language can be written with it.
The table has 151,936 rows; the tokenizer uses 151,665 of them. The rest are spare.
10 tokens in. One token out. In between, ideas 1 to 3 (the dot product, matrix multiplication, and softmax with attention) do the work; ideas 4 and 5 (surprise in bits, and the chain rule) are how it learned to.
Step 2: each token becomes a vector
A list of 1,024 numbers, and why it’s really an arrow.
Row 21,938 of Qwen3’s table is a list of 1,024 numbers. A list like that is called a vector.
With two numbers you can draw a vector as an arrow on paper: “go 3 right and 2 up.” With 1,024 numbers it’s still an arrow, just in a space with 1,024 separate axes to step along, where paper has only 2 (across and up). Nobody can picture that, me included. The good news: the math doesn’t care. The rules for lengths, angles and dot products (the next lesson) work the same way with any number of axes.
Nobody chose these numbers. They started out random, and training nudged them over and over until tokens that get used in similar ways ended up pointing in similar directions.
The map shows it: places huddle on one side, foods on another, and cat and dog sit together, apart from both. The map is only a sketch, so below is the check with the real vectors: how closely each word points the same way as Paris, using all 1,024 numbers.
Even France is far from 1. What matters is the gap: every place name scores between 0.32 and 0.47 with Paris, and every food scores 0.13 or less. The score is called cosine similarity: 1 means the two arrows point exactly the same way, and 0 means they’re at a right angle. It’s built from the dot product, the subject of the next lesson, where you’ll compute one yourself.
For the experts (skip it on a first read)
These are input embeddings (Qwen3-0.6B-Base ties them to the output layer). The 2D map is a principal component projection of the 12 vectors, each scaled to length 1; the first two components keep 35% of their variance. The cosines use all 1,024 dimensions.