Laboratory 03 · Verified
Connect score scaling and normalization to the attention pattern and resulting value mixture.
Interactive workbench
Follow one deterministic toy sequence from vectors to a weighted value mixture, then inspect any query-key relationship.
Start here
Attention answers that question for this fixed toy calculation. Each heatmap row is the word looking for context; each column is a word it may examine.
Each row is a small hand-authored vector for one token. These values make the arithmetic reproducible; they were not learned from language data.
X ∈ ℝ⁵ˣ⁴| Token | x₁ | x₂ | x₃ | x₄ |
|---|---|---|---|---|
| robot | 0.9 | 0.1 | 0.2 | 0.7 |
| follows | 0.2 | 0.9 | 0.6 | 0.1 |
| the | 0.1 | 0.2 | 0.1 | 0.3 |
| map | 0.8 | 0.3 | 0.9 | 0.4 |
| carefully | 0.3 | 0.8 | 0.2 | 0.9 |
| Query ↓ / Key → | robot | follows | the | map | carefully |
|---|---|---|---|---|---|
| robot | |||||
| follows | |||||
| the | |||||
| map | |||||
| carefully |
Read this cellWhile processing ‘follows’, this toy calculation assigns 20.4% of its attention to ‘map’.Fixed toy arithmetic, not learned language behavior.
Query row 2 · key column 4
Phase 01 · Input vectors. Selected follows query and map key. While processing ‘follows’, this toy calculation assigns 20.4% of its attention to ‘map’. Exact weight 0.2036. The selected row sums to 1.000000.
Read the calculation
Fixed matrices turn each toy token vector into a query, key, and value.
Query-key dot products are divided by the square root of the key width.
Softmax creates row-wise weights that combine the value vectors.