How LLMs Are Made / Glossary

Transformer

A transformer is the neural network design behind almost all modern language models. It processes every token of a text in parallel and uses attention to let the tokens share information.

The transformer was introduced in a 2017 research paper called “Attention Is All You Need”, first to translate between languages. Its key idea was to drop step-by-step reading and let every word look at every other word at once.

A transformer turns each token into an embedding, then passes it up through a stack of identical layers. In each layer, attention mixes in information from other tokens and a feed-forward part refines each token. The largest GPT-3 model stacks 96 of these layers.

Because the work happens in parallel, transformers train efficiently on modern chips, which made very large models possible. The same design is now also used for images, audio and code.