Skip to content
EN

Lesson 2 of 6 · 6 min · +100 XP

Where does the training data for large language models come from?

Large language models learn from a huge mix of text: public web pages collected by crawlers, books, scientific papers, open-source code, encyclopedias and forum posts, plus licensed and specially written data. Each source teaches something different, from facts to storytelling to programming. Modern models train on trillions of tokens, and the mix shapes what the model is good at.

▶ Try it

Why it matters

A model can only learn what is in its data. The mix of sources explains what it is good at, which languages it handles well and whose voices it echoes.

In 6 minutes you'll be able to

  • If a model is weak in your language or field, it probably saw little text about it during training.
  • Remember that public web text includes opinions and mistakes, so a model can echo both.
  • Look for a model's documentation: responsible makers describe their data sources.

What you'll practise

  1. Which pipe did it come from?

    Each snippet came from one source. Sort it into the right pipe.

    • Opening hours: Mon–Fri 9:00–17:00. Find us next to the station. → Web pages
    • Chapter 3. The rain hadn't stopped for three days when Mara reached the lighthouse. → Books
    • def area(r): return 3.14159 * r * r → Code
    • Re: disk full error? Clearing the cache fixed it for me, thanks all! → Forums
    • “I never meant to leave,” she whispered, closing the old diary. → Books
    • for (let i = 0; i < 10; i++) { total += i; } → Code
    • Q: Is watering cacti weekly OK? A: Too often in winter, try monthly. → Forums
    • 10 easy tips for growing tomatoes on a balcony → Web pages

  2. What did each pipe teach?

    Each line comes in the style of one source. Pick the next word, then see Vireo's prediction. The percentages are illustrative.

    1. It was a dark and stormy ___

      • night: 90% (most likely)
      • day: 3%
      • morning: 2%
      • sandwich: 0%
      • other words: 5%
      A famous opening line, repeated in countless books and articles. Books teach storytelling and well-known phrases.

    2. if user is None:
      return       ___

      • None: 55% (most likely)
      • False: 20%
      • user: 5%
      • hello: 0.1%
      • other words: 20%
      After reading millions of lines of public code, Vireo knows what usually follows. That's where its coding skill comes from.

    3. Has anyone else had this problem? EDIT: fixed it, ___

      • thanks: 40% (most likely)
      • turns: 20%
      • never: 2%
      • other words: 38%
      Forum threads teach the back-and-forth of asking for help and sharing fixes, useful for an assistant.

    4. We're open Monday to ___

      • Friday: 70% (most likely)
      • Saturday: 15%
      • Sunday: 5%
      • other words: 10%
      Ordinary web pages teach how the everyday world is described: opening hours, prices, directions.

  3. Plan Vireo's data mix

    Tomás drafted a plan for the next training run. Switch each line to the setting that makes a better model.

    • Add text in many languages
    • Copy the most popular page 1,000 times so it counts more
    • Add public code from open-source projects
    • Add people's private messages
    • Add textbooks and science papers we have permission to use
    • Use text from just one big website

Remember

A model only knows what it has read. Its mix of web pages, books, code and conversations shapes everything it does.

A model is shaped by its data mix: web pages, books, code and conversations each teach something different.

Key facts

  • Common Crawl, a non-profit, has archived billions of web pages since 2008 and makes them freely available; filtered versions feed many language models. · Source: Common Crawl Foundation
  • GPT-3's training mix gave about 60% of its weight to filtered web crawl text, 22% to curated web pages, 16% to two book collections and 3% to Wikipedia. · Source: Brown et al., OpenAI, Language Models are Few-Shot Learners, 2020
  • The Pile, an open dataset for training language models, combined 22 different sources, from scientific papers to code and forum posts. · Source: Gao et al., EleutherAI, The Pile, 2020
  • Llama 3 was pretrained on over 15 trillion tokens collected from publicly available sources. · Source: Meta, Introducing Meta Llama 3, 2024

Updated:

Questions people ask

What data are LLMs trained on?

Mostly text: public web pages, books, scientific papers, encyclopedias, open-source code and forum discussions, often plus licensed data and examples written by people specially for training.

How much data is used to train an LLM?

Modern models are pretrained on trillions of tokens, far more text than a person could read in many lifetimes. The exact amount depends on the model.

Do LLMs train on books?

Many do, since books offer long, well-edited writing. Which books may be used, and on what terms, is debated and the subject of lawsuits in several countries.

Why does the data mix matter?

A model learns the patterns in what it reads. More code makes it better at programming, more languages make it more multilingual, and one-sided data makes it one-sided.

Related terms