How LLMs Are Made / Glossary

Context window

A context window is the maximum amount of text, counted in tokens, that a language model can take in at once, including your messages, its replies, pasted files and hidden instructions.

Everything the model uses to write its next token has to fit in the window. When a conversation grows past it, the oldest parts no longer fit and the model can't use them. It also doesn't remember earlier chats unless an app feeds them back in.

Windows have grown fast: GPT-3 could read 2,048 tokens at once, and Llama 3.1 supports 128,000, long enough for a whole book. They are still limited, partly because the work of attention grows with the square of the text's length.

A bigger window isn't always used well. Researchers found that models use information at the start and end of a long context better than information buried in the middle, so put what matters most where it stands out.