Skip to content
EN

Lesson 5 of 6 · 6 min · +100 XP

What is a context window, and why does a chatbot forget?

A context window is the fixed amount of text, counted in tokens, that a language model can see at once: your messages, its replies, pasted files and hidden instructions. When a conversation grows past it, the oldest parts no longer fit and the model can't use them. It doesn't remember earlier chats unless an app feeds them back in.

▶ Try it

Why it matters

A model has no memory between messages: it rereads the whole conversation every time, and only as much as fits its window. That's why long chats lose early details.

In 6 minutes you'll be able to

  • The window holds your messages, the model's replies, pasted files and the app's instructions.
  • When a chat gets too long, the oldest parts fall out and the model can't use them.
  • In a long chat, restate key details or start fresh with a short summary.

What you'll practise

  1. What's in the window?

    Each time you send a message, the model reads its window from scratch. Which of these take up room in it?

    • Your messages in this chat
    • Vireo's own replies in this chat
    • A document you paste in
    • Hidden instructions from the app
    • Every book Vireo was trained on
    • A chat you had with Vireo last month

  2. Pack the window

    Play Lina's chat and watch the window slide. Then help Vireo answer. Token counts are rounded, made-up examples.

    What Vireo can see: 600 · Vireo's window holds 600 tokens. Press Play the chat and watch what happens when the conversation outgrows it.

    1. Lina: Help me plan a surprise party for my sister Maya! It's on Saturday 12 April, and she's allergic to peanuts. (40 tok)
      summary: Party: Sat 12 April. Maya is allergic to peanuts. (14 tok)
    2. Vireo: How fun! Let's start with a guest list and a place. (30 tok)
      summary: Asked about guests and a place. (8 tok)
    3. Lina: guest-list.txt: 24 names, phone numbers and who brings what (520 tok)
      summary: Guest list: 24 people. (8 tok)
    4. Vireo: With 24 guests, a community hall or a big garden would both work. (60 tok)
      summary: Hall or garden for 24. (9 tok)
    5. Lina: Sorry, my cat just walked across the keyboard 😹 (20 tok)
      summary: Cat on the keyboard. (5 tok)
    6. Vireo: Here's a detailed comparison of the hall and the garden: price, space, weather, parking… (300 tok)
      summary: Lina picked the community hall. (8 tok)
    7. Lina: For the cake: lemon sponge with raspberry, her favourite! (25 tok)
      summary: Talked about the cake. (6 tok)
    8. Vireo: Playlist ideas: 40 songs from the 90s, sorted by mood… (280 tok)
      summary: Playlist: 90s songs. (6 tok)
    9. Lina: Now write the cake order for the bakery: date, number of guests, flavour and allergies. (30 tok)

    ✗ Vireo: Cake order: one lemon sponge with raspberry, for about 10 guests. Pick-up date: not sure. Allergies: none mentioned.

    The date, the allergy and the guest count were all in early messages that slid out of the window. Vireo wasn't careless: it simply couldn't see them any more.

    ✓ Vireo: Cake order: lemon sponge with raspberry for 24 guests, to collect on Saturday 12 April. Strictly peanut-free, please: the guest of honour is allergic.

    Everything fits, and every detail the bakery needs survived. That's what apps do behind the scenes in long chats: trim, summarise and keep what matters.

  3. Window or training?

    Where does Vireo get this from: its context window, its training, or neither?

    • Your name, which you told it ten messages ago → Context window
    • That water boils at 100 °C at sea level → Training
    • The PDF you uploaded a minute ago → Context window
    • What you asked it in a different chat last week → Neither
    • How to write a polite email → Training
    • This morning's news, with no web search → Neither

Remember

A model only sees what fits in its context window. Long chats push early details out, so repeat or summarise what matters.

The context window is the model's whole short-term memory. What doesn't fit, it can't see.

Key facts

  • GPT-3, one of the first very large language models, could read a context window of 2,048 tokens at once. · Source: Brown et al., Language Models are Few-Shot Learners, OpenAI, 2020
  • Meta's Llama 3.1 models support a context window of 128,000 tokens, long enough for a whole book. · Source: Meta, The Llama 3 Herd of Models, 2024
  • In a transformer, every token attends to every other token, so the work of attention grows with the square of the text's length. That's one reason windows are limited. · Source: Vaswani et al., Attention Is All You Need, Google, 2017
  • Researchers found that models use information at the start and end of a long context better than information buried in the middle. · Source: Liu et al., Lost in the Middle: How Language Models Use Long Contexts, 2023

Updated:

Questions people ask

What is a context window in AI?

It's the maximum amount of text, measured in tokens, that a language model can take in at once. It includes your messages, the model's replies, uploaded files and the app's hidden instructions.

Why does a chatbot forget what I said earlier?

If a conversation grows longer than the context window, the oldest messages no longer fit and the model can't see them. Repeating key details or starting a new chat with a summary helps.

Do AI chatbots remember previous conversations?

The model itself doesn't: each chat starts with an empty window. Some apps have a memory feature that saves notes and puts them back into the window in later chats.

Is a bigger context window always better?

It lets the model read longer documents and chats, but it costs more computing power, and models can still overlook details buried in the middle of very long texts.

Related terms