“The battery is…” what?
A language model receives a partial sentence and must turn its current context into a ranked set of possible next tokens. The useful question is not “what does it know?” but “what state does this sentence produce?”
Back to the lesson shelf
Trans
LESSON 07 The full forward pass
Trans
former.
From a sentence to the next token: the whole relay, with every handoff left visible.
In one lineTokens → context → transformed state → a next-token distribution.
FORWARD_PASS / 07 running
sequenceprediction
read → mix → transform → choosetoy state / live
Short version: a transformer keeps rewriting the sentence until the next token has a useful context.Step, change, compare.
00 Overview / start here
What comes after attention?
You have seen the ingredients: neural networks transform signals, embeddings turn tokens into vectors, and attention mixes context. Now follow them as one causal journey.
A context-rewriting relay.
A transformer repeatedly lets tokens exchange information, transforms the resulting vectors, and finally converts the last state into scores for the vocabulary.
the before / after storysame words → richer state
before / raw sequence
Thebatteryis
Pieces arrive as IDs plus positions. “is” is still mostly a local token.
after / prediction-ready
Thebatteryischarged
The final state carries clues about subject, relation and likely continuation.
the complete journeyinput → operation → output
- 01 / representText becomes tokens.
Split a sentence into discrete pieces, then look up a small vector for each piece and add position.
- 02 / relateAttention mixes context.
Each position asks which other positions are useful, then carries a weighted mixture forward.
- 03 / predictBlocks make scores useful.
Feed-forward transformations repeat across depth. Final logits become probabilities for the next token.
A relay team, not a tiny reader.
Each block receives a baton containing the current numeric state, improves or rearranges it, and passes it onward. The limit: a transformer has no human understanding hidden inside the analogy; it calculates with learned numbers.
01 Run the forward pass
One sentence. Eight handoffs.
Choose a prompt, then move one stage at a time. Every panel is a toy snapshot of the same state, updated as the sentence travels.
Use the inspector as your guide.
At each stop, read the three-part sentence: what enters, what operation happens, and what leaves. Auto-run is available, but stepping slowly makes the causality easier to see.
forward-pass stages / click a stop
→→→→→→→
current state / textA partial sentence enters.
the signal is ready to move
stage 01text is the starting signal
The model does not leap from words to an answer. It keeps passing a better state forward.
02 Read the timeline
Eight states, one direction.
The names can feel abstract when listed in a glossary. Here they are ordered as a working pipeline: each output becomes the next input.
Zoom out from the simulator.
Select a timeline state to highlight the corresponding handoff. The small values repeat the simulator’s current toy prompt.
causal map / selected promptThe battery is
01 / textStart with the prompt.
A text string is the only thing the user sees. Everything else is a numeric transformation of this starting point.
text→frame→tokenizable sequence
A transformer is a direction, not a mystery box. Every stage leaves a handoff.
03 Why stack blocks?
Depth gives the state more chances.
One block can mix context and transform a representation. Repeating that block lets later passes build on earlier passes, gradually making the state more useful for the prediction.
Run the baton through depth.
Change the number of repeated blocks. The signal is still a toy vector, but the visible trace shows what “more depth” means operationally.
three passes / richer combination
Depth is not a bigger dictionary. It is more rewrites of the same moving state.
04 Keep attention in view
Context is a weighted mix.
Attention is the communication step inside each transformer block. A token asks for useful context, and the available tokens contribute in different proportions.
Who does the final token consult?
This compact map uses the final token as the query. Switch prompts and the toy weights change with the subject.
queryis
The final token leans on the subject noun before it.100% visible mixture
Attention chooses what enters the mixture. The block decides what to do with it.
05 Do the small math
One weighted sum. Then a choice.
The production operation is high-dimensional, but the logic can fit in a few lines. Here is one visible calculation that matches the toy attention mixture.
Calculate before you generalize.
Use the displayed weights and values as ordinary arithmetic. The result is a new vector, not a word or a thought.
input values
The[0.20, 0.30]× 0.18
battery[0.80, 0.60]× 0.62
is[0.40, 0.50]× 0.20
weighted sum0.18 × [0.20, 0.30]
+ 0.62 × [0.80, 0.60]
+ 0.20 × [0.40, 0.50]≈ [0.59, 0.55]
+ 0.62 × [0.80, 0.60]
+ 0.20 × [0.40, 0.50]≈ [0.59, 0.55]
what leaves attention[0.59, 0.55]
The mixture is mostly the subject’s value, with a trace of the surrounding tokens. A feed-forward transformation can now reshape this result.
weights add to 1.00 / output is a vectorillustrative dimensions only
minimal runnable code / JavaScriptcopy the mechanism, not the scale
const values = {
The: [0.20, 0.30],
battery: [0.80, 0.60],
is: [0.40, 0.50],
};
const weights = [0.18, 0.62, 0.20];
const mixed = [0, 1].map((dimension) =>
weights.reduce((sum, weight, index) =>
sum + weight * values[Object.keys(values)[index]][dimension], 0
)
);
console.log(mixed.map((value) => value.toFixed(2)));
// ["0.59", "0.55"]This runnable sketch shows the attention output only. Production transformers hide vocabulary lookup, positional signals, learned Q/K/V projections, multiple heads, normalization, residual paths, matrix multiplication, batching and much larger vectors behind the same causal outline.
The scary-looking model is built from familiar moves. Lookups, sums, transforms, scores.
06 Consolidate the chain
Predict before you click.
Return to the first prompt. Before stepping to the final state, make a quiet prediction: which next token should lead, and which upstream clue made it plausible?
pause / retrieve the causal story“The battery is”
Try to name the route without looking: text becomes tokens; tokens get vectors and positions; attention mixes context; repeated blocks transform the state; logits become probabilities; one next token is selected.
Reveal one illustrative continuation
toy top choicecharged / 62%
That number is invented for this lesson. The important explanation is structural: the final probability comes from the transformed state, and the transformed state depends on every earlier handoff.
01Represent
Text → tokens → vectors.
02Relate
Attention mixes useful context.
03Rewrite
Blocks refine the state.
04Choose
Softmax ranks next tokens.
LIMITATIONS / KEEP THE MODEL HONEST
This is a map of operations, not a window into “what the model thinks.”
The toy numbers are not model outputs, attention weights are not guaranteed explanations, and a next-token probability is not a statement of truth. Real systems add scale, training data, architecture details and failure modes that this field guide intentionally leaves out.
PRIMARY SOURCES
The original architecture: Attention Is All You Need. For a friendly visual companion: The Illustrated Transformer by Jay Alammar.
NEXT STEP
Revisit the ingredients, then return to the shelf and choose the next question worth making visible.
FAQ Transformers / quick answers
Keep the
relay moving.
One last set of plain answers after the whole forward pass.
01 / forward passHow does a transformer generate the next token?
It turns text into tokens and vectors, mixes context with attention, refines the state through blocks, converts the final state into vocabulary scores, and selects a next-token candidate.
02 / outputWhat is the difference between logits and probabilities?
Logits are raw relative scores for candidate tokens. Softmax converts those scores into probabilities that share one scale and add up to 100 percent.
03 / limitsIs this visualization a real model output?
No. The values are deliberately tiny teaching numbers. They expose the causal order of the operations without pretending to reproduce a production model’s weights or prediction.