LESSON 06 The repeatable unit
The small circuit a transformer runs again and again: mix context, preserve a route, stabilize the numbers, then enrich the signal.
00 Overview / start here
A language model needs to keep a useful thread of meaning while it adds context and new features. The transformer block is the repeatable unit that does this work.
In “The robot carried the battery because it was heavy,” the token “it” starts with a rough representation. It needs the surrounding words, but the next calculation should not erase what “it” already carried.
It lets tokens exchange information with self-attention, adds the previous signal back through a residual path, stabilizes the scale with normalization, and then applies a small feed-forward network.
For this lesson, “it” is a four-number vector. The vector is a calculated state; the embedding table that produced it is learned.
Attention mixes information across positions. The feed-forward network works on each position separately to create richer features.
Residual additions keep a direct route from earlier values. The output can enter the next identical block, many times.
Attention pulls in a little “battery” and “heavy” information. The first residual adds the original “it” signal back. The FFN notices a feature combination, the second residual keeps both, and normalization keeps the values workable.
01 Trace the block
The names are less important than the sequence. Move stage by stage and watch the same four-number signal change. Use Previous, Next or the arrow keys when the stepper has focus.
This is a toy trace for the token “it.” Values are rounded to two decimals. The attention mix and FFN weights are illustrative learned parameters; residual addition and normalization are calculated from the current state.
A token begins as a row of learned numbers. Nothing has mixed yet.
A block does not replace the signal in one leap. It edits the signal in careful passes.
02 Run the skip-path experiment
A residual connection is an element-wise addition: line up two vectors and add their matching coordinates. Toggle it to see what the block loses when the direct route disappears.
The gauge below is deliberately simple: it shows how much of each original component is present in the first sum. It is not a probability and does not claim that a real model has a literal percentage of “old meaning.”
Each coordinate receives the old value and the new value. The block can add context without forcing the old route to vanish.
The shortcut is not laziness. It is a memory lane.
03 Repeat the unit
One block can make a small update. A stack of identical-shaped blocks can make a representation progressively richer. Change the depth and compare the toy states.
This control repeats the same illustrative block. It is not a trained language model: the weights stay fixed, the dimensions stay tiny, and the changing vectors only demonstrate repeated transformation.
| pass | vector state | what can change |
|---|
Depth is not a new kind of block. It is another chance to transform the same kind of state.
04 Do the small math
Now pin the moving picture to one explicit calculation. This makes the boundary clear: learned weights are parameters; the vectors that pass through the block are calculated states.
The left side is static so you can read it slowly. The right side mirrors the same numbers and labels the shapes, operation order and value origin.
x = [0.70, 0.20, 0.80, 0.10]0.15V₁ + 0.65V₂ + 0.20V₃
= [0.25, 0.47, 0.23, 0.54]x + attention
= [0.95, 0.67, 1.03, 0.64](z − mean(z)) / √(variance + ε)
≈ [0.73, −0.91, 1.23, −1.05]FFN(norm) ≈ [0.10, 0.15, −0.07, 0.16]
output ≈ [0.81, −0.92, 1.17, −1.06]05 Run the tiny version
A minimal JavaScript example performs one toy block in the same order as the diagrams. It runs in this page with no build step or library.
The code keeps the learned toy weights visible. Everything printed after `embedding` is calculated from them and from the current vector.
const embedding = [0.70, 0.20, 0.80, 0.10];
const attentionWeights = [0.15, 0.65, 0.20]; // toy learned scores
const values = [[0.20, 0.10, 0.40, 0.30], [0.30, 0.60, 0.20, 0.70], [0.10, 0.30, 0.20, 0.20]];
const W1 = [[0.60, 0.10, -0.20, 0.00], [-0.10, 0.40, 0.20, 0.30], [0.20, -0.30, 0.40, 0.10]];
const W2 = [[0.20, 0.00, 0.10], [-0.10, 0.10, 0.20], [0.10, 0.20, -0.10], [0.00, 0.30, 0.20]];
const add = (a, b) => a.map((value, i) => value + b[i]);
const mix = values[0].map((_, i) => values.reduce((sum, value, j) => sum + attentionWeights[j] * value[i], 0));
const normalize = (vector) => { const mean = vector.reduce((s, v) => s + v, 0) / vector.length; const variance = vector.reduce((s, v) => s + (v - mean) ** 2, 0) / vector.length; return vector.map(v => (v - mean) / Math.sqrt(variance + 0.0001)); };
const relu = value => Math.max(0, value);
const firstResidual = add(embedding, mix);
const stable = normalize(firstResidual);
const hidden = W1.map(row => relu(row.reduce((sum, weight, i) => sum + weight * stable[i], 0)));
const ffn = W2.map(row => row.reduce((sum, weight, i) => sum + weight * hidden[i], 0));
const output = normalize(add(stable, ffn));
console.log({ embedding, mix, firstResidual, stable, hidden, ffn, output });No run yet. The output will match the calculation above.
What this omits: batching, positional details, multiple attention heads, dropout, masking and training. Those are important production details, but hiding them here keeps the block’s causal order inspectable.
06 Consolidate / try this
Close the loop without memorizing the labels. Use the experiments above as evidence, then make one prediction.
Turn the residual connection off. Before you look at the output, predict what happens to the original vector’s component share and why the later normalization still runs.
Replace the attention weight for “robot” with 0.80 and lower the “The” weight. Which part of the attention mix gets louder? Which residual coordinates change?
Embeddings start the state. Attention mixes positions. Residuals preserve a direct route. Normalization keeps scale manageable. FFN enriches each position. The same shape can enter another block.
This field guide uses one token position, four features, fixed toy parameters and a simplified post-attention order. Real LLMs add positional information, multiple heads, causal masks, layer-norm variants, efficient kernels, batching, training updates and many more blocks. Those details refine the implementation; they do not change the core story traced here.
source note The original Attention Is All You Need paper introduced the Transformer architecture. This page’s vectors, weights and outputs are deliberately tiny teaching values. For normalization background, see the original Layer Normalization paper.
FAQ Transformer blocks / quick answers
Keep the reusable unit clear before the full transformer story gets larger.
A transformer block is a repeatable unit that lets positions exchange context through attention, updates features with a feed-forward network, and uses residual paths and normalization to keep the signal workable.
A residual connection adds an earlier vector back to a transformed vector element by element. It gives the block a direct route for preserving useful information while adding a new update.
Each block can refine the representation it receives. Repeating the same overall shape lets later blocks work with a state that already contains more context and feature transformations.