What does “it” refer to?
In “The robot carried the battery because it was heavy,” the word “it” needs the words around it. The sentence is the clue.
Back to the lesson shelf
Atten
LESSON 04 The context spotlight
Atten
tion.
A weighted spotlight that helps a model decide which nearby tokens matter right now.
In one lineAsk a question → weigh the context → mix the useful parts.
CONTEXT_SPOTLIGHT / 04 focusing
querycontext
query → scores → mixturetoy weights / live
Short version: one token asks, the context answers.Click, focus, compare.
00 Overview / start here
Meaning lives in the surroundings.
A word can be ambiguous on its own. Attention gives a model a way to look around, decide what matters, and carry a useful blend forward.
A weighted spotlight.
Attention lets a model compare one token with the available context, give each context token a weight, and mix the useful information.
the basic storyinput → operation → output
- 01 / inputA query token asks.
The current token, such as “it,” is the question: what information would help me here?
- 02 / operationContext gets weighted.
Scores are normalized into percentages. A larger percentage means more of that token enters the mixture.
- 03 / outputA mixed signal comes out.
The model carries a new representation made from the weighted values, ready for the next calculation.
70% + 20% + 10% = 100%
If “battery” gets 70%, “robot” 20%, and “heavy” 10%, the mixed signal is mostly battery information, with smaller contributions from the other clues.
01 Spotlight on a sentence
Which words help right now?
Choose a sentence, then click any token. The toy model treats that token as the query and redistributes attention across the available context.
Follow the weights.
Every context bar is a visible percentage. The bars share one whole: the visible weights always add to 100%.
choose a sentence
sentence tokens / click a query
attention map / model toy100% visible context
The spotlight is strongest on the clue that makes “it” understandable.
Attention is not a spotlight because the model “sees.” It is a calculated mixture.
02 Change the focus
Spread out or lock on.
The focus control changes how concentrated the illustrative weights are. The causal-mask mode then hides tokens that would arrive later in a left-to-right prediction.
Turn one knob, watch the mixture move.
A low focus spreads attention across more context. A high focus concentrates it on the strongest clue. This is a visualization of the effect, not a training control.
next-word mode
what changedbalanced spotlight
The useful clue is bright, but nearby context still contributes to the mixture.
focus 50 → normalize scores → mixed output
causal maskfuture tokens are available
In a normal full-context view, the query can compare tokens on both sides of it.
A mask changes the available evidence. Unavailable context gets no percentage.
03 Three jobs
Query. Key. Value.
These three names describe jobs in the attention operation. Step through them to see how a question becomes a weighted mixture.
Trace one attention calculation.
Select a job. The diagram highlights the part currently doing the work, and the number panel shows the matching intermediate state.
Qitthe question
compares with
Kbatteryphysical object
Krobotagent
Kheavyproperty
selects
V[0.8, 0.1]object signal
V[0.2, 0.7]agent signal
V[0.1, 0.2]property signal
The letters are jobs, not three extra words to memorize. Ask, match, mix.
04 Do the small math
Percentages become a new signal.
Weights only say how much each source contributes. The values supply the information that is actually mixed.
See input → weight → output.
The vectors below are two-number toy values. Drag focus in Experiment 02 or use this panel’s control to see the mixture change.
battery / value[0.80, 0.10]70%
+
robot / value[0.20, 0.70]20%
+
heavy / value[0.10, 0.20]10%
weighted mixture / output[0.61, 0.23]
70% × [0.80, 0.10] + 20% × [0.20, 0.70] + 10% × [0.10, 0.20]
At balanced focus, battery information dominates while robot and heavy still leave a trace.
A weighted sum is not a vote with a winner. It is a blend with visible proportions.
TRY IT YOURSELF / CONSOLIDATE
Predict the spotlight.
Before revealing the hint, switch Experiment 01 to “fragile,” keep the query on “it,” and raise focus. Which context token should take the largest share, and why?
“battery” should still be important, but “fragile” is the strongest material clue in this toy sentence. A higher focus makes that clue take a larger share of the visible mixture.
PRIMARY SOURCE The original transformer paper introduces attention as a way to relate positions in a sequence: Attention Is All You Need. These attention weights are a teaching visualization, not a guarantee of human-readable reasoning.
FAQ Attention / quick answers
Look around
first.
A few plain answers for the most common questions about the spotlight.
01 / definitionWhat is attention in AI?
Attention is a calculation that gives context tokens different weights, then combines their value vectors into a new signal for the current position.
02 / QKVWhat do Query, Key and Value mean?
The query describes what the current position is looking for, keys help compare available positions, and values carry the information that gets mixed after scoring.
03 / limitsDoes an attention map show the model’s reasoning?
Not by itself. An attention map shows one part of a computation. It can make data flow easier to inspect, but it is not a guaranteed explanation of a model’s internal reasoning.