CertScroll

Attention mechanism

The mechanism that lets a model weigh every token it is allowed to see in the input when building the representation for one token, rather than only the tokens immediately before it. In a decoder-only model that means every earlier token, since causal masking hides the future; an encoder-only model like BERT can see the whole sequence in both directions. Multiple attention heads run in parallel, each free to learn a different kind of relationship, which is what lets a model track a pronoun back to its subject across a long paragraph, regardless of how the text was split during tokenization.