Skip to content
EN

Lesson 1 of 6 · 6 min · +100 XP

What is a transformer model in AI?

A transformer is the neural network design behind modern language models. It turns each token into a list of numbers, then passes them up through a stack of identical layers. In every layer, attention lets tokens share information and a feed-forward part refines each one. At the top, it scores every possible next token.

▶ Try it

Why it matters

Almost every modern language model is a transformer. Knowing its two moves, sharing between words and thinking about each word, explains how a model builds up meaning layer by layer.

In 6 minutes you'll be able to

  • Text becomes tokens, and each token becomes a list of numbers before entering the stack.
  • Every layer has two parts: attention, where words share information, and feed-forward, where each word is refined.
  • Stacking dozens of layers turns rough guesses into sharp predictions.

What you'll practise

  1. Send a sentence up the stack

    Put the journey of “The sky is very” in order, from raw text to Vireo's guess.

    1. The text is cut into tokens
    2. Each token becomes a list of numbers, its embedding
    3. Attention: each token gathers clues from earlier tokens
    4. Feed-forward: each token's numbers are refined on their own
    5. Those two parts repeat in every layer, dozens of times
    6. At the top, the last token's numbers become a score for every possible next token

  2. Peek inside the tower

    Engineers can read out Vireo's guess halfway up the stack. Each round is the same sentence at a different height. The percentages are illustrative.

    1. Layer 2 of 48:
      The keys to the old cabinet       ___

      • is: 34% (most likely)
      • are: 20%
      • door: 10%
      • was: 8%
      • other words: 28%
      Low in the stack, the guess leans on the nearest word, “cabinet”, which is singular. The layers haven't linked the verb to “keys” yet.

    2. Layer 24 of 48:
      The keys to the old cabinet       ___

      • are: 45% (most likely)
      • is: 24%
      • were: 10%
      • was: 5%
      • other words: 16%
      Halfway up, attention has connected the verb to “keys”, so plural verbs overtake “is”.

    3. Layer 48 of 48:
      The keys to the old cabinet       ___

      • are: 78% (most likely)
      • were: 10%
      • is: 3%
      • was: 1%
      • other words: 8%
      At the top the guess is sharp. Each layer added a little understanding, which is why transformers stack so many of them.

  3. Which part does this?

    Every layer has two parts. Sort each job.

    • “it” collects information from “cat” three words back → Attention: words share
    • Works on one token at a time, without looking at the others → Feed-forward: each word alone
    • Links a verb to its subject several words earlier → Attention: words share
    • Holds much of what the model has stored, like facts it absorbed → Feed-forward: each word alone
    • Decides how much each earlier word matters right now → Attention: words share

  4. Why did transformers win?

    1. Older language models read text one word after another. What did the transformer change?

      • It handles all the tokens of a text side by side, so training uses chips efficiently (Answer)
      • It reads even more slowly and carefully, one letter at a time

    2. Vireo has 48 layers. How do they differ from each other?

      • Same design, but each layer has its own learned weights (Answer)
      • Each layer is a completely different kind of machine

Remember

A transformer is one layer design stacked many times: attention lets words share, feed-forward refines each word, and the top scores the next token.

Tokens rise through dozens of identical layers. Each layer shares information between words, then refines each word.

Key facts

  • The transformer was introduced in 2017, first to translate between languages. · Source: Vaswani et al., Attention Is All You Need, Google, 2017
  • The original transformer stacked 6 layers on each side, one side to read the input and one to write the output. · Source: Vaswani et al., 2017
  • The largest GPT-3 model stacks 96 transformer layers. · Source: Brown et al., Language Models are Few-Shot Learners, OpenAI, 2020
  • Researchers found that the feed-forward parts of transformer layers behave like key-value memories, storing patterns from the training text. · Source: Geva et al., Transformer Feed-Forward Layers Are Key-Value Memories, 2021

Updated:

Questions people ask

What does the T in GPT stand for?

Transformer. GPT stands for Generative Pre-trained Transformer: a model that generates text, is first pretrained on lots of text, and uses the transformer design.

What is a layer in a neural network?

One stage of processing. In a transformer, each layer takes every token's numbers, lets them share information through attention, refines them with a feed-forward part, and passes them up to the next layer.

Why are transformers better than older models?

They process all the tokens of a text in parallel during training and can link words that are far apart. That made it practical to train much bigger models on much more text.

Are all large language models transformers?

Almost all of today's popular ones are, or are close variants. Researchers keep testing other designs, but the transformer has been the standard since around 2018.

Related terms