How LLMs Are Made / Glossary

Attention mechanism

The attention mechanism is the part of a transformer that lets each token weigh how much every other token matters to it, then take in a mix of their information based on those weights.

For every token, attention scores how relevant each other token is, turns the scores into weights that add up to 100% and blends in their information. That is how “it” in a sentence can find the noun it refers to, and how “bank” learns whether it means money or a river.

A model runs several attention heads side by side, and each can learn a different kind of link, such as pronouns to nouns or verbs to their objects. The original transformer used 8 heads per layer; GPT-3 uses 96.

Attention compares every token with every other token, so its work grows with the square of the text's length. That is one main reason a model's context window has a limit.