Skip to content
EN

Lesson 6 of 6 · 6 min · +100 XP

What is a base model in AI?

A base model is a language model straight out of pretraining. It has learned to continue text, not to follow instructions, so a question often gets more questions back instead of an answer. It knows a lot, but you only get answers if your text looks like a document where an answer comes next. Assistant training comes later.

▶ Try it

Why it matters

After months of pretraining you get a model that knows a lot but doesn't help. Seeing why explains what all the later training is for, and why prompts work the way they do.

In 6 minutes you'll be able to

  • Pretraining produces a base model: a powerful autocomplete, not an assistant.
  • To get useful output from a base model, write the start of a document where your answer comes next.
  • Instructions like “please” or “you are helpful” only work once a model has had assistant training.

What you'll practise

  1. Meet the base model

    You type into Vireo's raw base model. Choose what you do at each step.

    Vireo (base) · pretrained only

    • Base model loaded. It continues whatever text you give it.

    1. You want to learn something. What do you type?

      • How do plants grow? (Answer)
        That's a base model. It saw what looks like a list of questions, so the likeliest next text was… more questions.
        • You: How do plants grow?
        • Vireo (base): How do birds fly?
          How do fish breathe?
          Why is the sea salty?
      • Write a short poem about the sea. (Answer)
        It took your request for the first line of a worksheet and kept writing the worksheet.
        • You: Write a short poem about the sea.
        • Vireo (base): Write a short story about a dragon.
          Write a letter to your future self.
          Homework is due on Friday.

    2. It didn't answer you. Why not?

      • It continues documents. Nobody has taught it to follow requests yet (Answer)
        Exactly. Pretraining only taught it to predict the next token of web-like text. Answering is a skill that comes later.
      • It's broken and needs restarting
        Almost. It works perfectly: it does exactly what pretraining trained it to do, which is continue text.
      • It has never read anything about the topic
        Almost. It has read plenty about it. It just doesn't know that it is supposed to answer you.

    3. Can you still get an answer out of it? Make the answer the likeliest next text.

      • PLEASE answer my question!!
        Almost. Pleading is just more text to continue, so it may well reply with more pleading.
      • Start a Q&A page: “Q: How do plants grow? A:” (Answer)
        On a Q&A page, an answer is what comes after “A:”. Notice it then starts the next question: it's still just continuing the page.
        • You: Q: How do plants grow?
          A:
        • Vireo (base): Plants use sunlight, water and air to make their own food. This is called photosynthesis.
          Q: Why are leaves green?
    • Turning this autocomplete into an assistant that simply answers is the next stage of training.

  2. Build a document it will finish

    Tap chips to add them to the document and watch what the base model writes next. Not every chip helps.

    1. You want the base model to explain why the sky is blue.

      Your document: why is the sky blue

      Vireo (base) continues:

      • why is the sky blue at night
      • why is the sea blue
      • why is my sky blue in the game
      • Rewrite it as “Q: Why is the sky blue? A:” ( Pattern) → A: Sunlight contains every colour.
        Better. The Q-then-A pattern makes an answer the likeliest next text.
      • Put one answered question above it ( Example) → Air scatters blue light much more than red,
        Better. An example shows the model exactly what kind of text comes next: a clear answer.
      • Title the page “Science questions, answered by teachers” ( Frame) → so blue reaches your eyes from all over the sky.
        Better. A frame sets the kind of document. Teacher answers are clear and correct, so the continuation is too.
      • Add “Please answer this, thank you!” ( Request) · taken out
        Almost. A base model doesn't follow requests. “Please” is just more text to continue.
      • Add “You are a helpful assistant.” ( Request) · taken out
        Almost. That line works on trained assistants. To a base model it's just another sentence in the document.

    2. Now you want a two-line poem about the sea.

      Your document: poem about the sea

      Vireo (base) continues:

      • poem about the moon
      • poem about the sea for kids free
      • poem generator online
      • Start a poetry book page: “Sea Poems, page 12” ( Frame) → The Tide
        Better. Inside a poetry book, a poem is the likeliest next thing: first its title.
      • Paste a short poem about rain above it ( Example) → The waves come in, the waves go out,
        Better. The example sets the length and style, so the model copies the pattern: two short rhyming lines.
      • End with a title and a line break, ready for the poem ( Pattern) → they whisper what the sea's about.
        Better. Stopping right where a poem starts leaves only one natural continuation: the poem.
      • Add “Make it amazing!!!” ( Request) · taken out
        Almost. Hype words don't change what kind of document this is, so the continuation stays the same.
      • Add “Do NOT write a list of searches” ( Request) · taken out
        Almost. Instructions don't steer a base model. It may even continue with more rules.

  3. Base model or assistant?

    1. Which reply came from a base model?

      • You ask “Best pizza topping?” and get “Best burger topping? Best ice cream flavour?” (Answer)
      • You ask “Best pizza topping?” and get “Tastes differ, but mushrooms are a popular choice.”

    2. So what does Vireo's base model already have, and what is missing?

      • It has lots of knowledge and language skill, but not the habit of helping (Answer)
      • It has nothing useful yet, all the knowledge comes later

Remember

A base model continues documents. It knows a lot, but it only answers when an answer is the likeliest next text. Assistant training comes next.

A base model continues documents. To get an answer, make the answer the most likely next text.

Key facts

  • GPT-3 showed that a pretrained model could do new tasks from a few examples placed in the prompt, without any retraining, a technique called few-shot prompting. · Source: Brown et al., Language Models are Few-Shot Learners, OpenAI, 2020
  • Human labelers preferred answers from a 1.3-billion-parameter model trained to follow instructions over answers from the 175-billion-parameter GPT-3 base model. · Source: Ouyang et al., Training language models to follow instructions with human feedback, OpenAI, 2022
  • Many labs release both versions of a model: the base model for researchers to build on, and an instruction-tuned version for chatting.
  • Most of a model's knowledge comes from pretraining. The later assistant training uses far less data and mainly shapes how the model behaves.

Updated:

Questions people ask

What is the difference between a base model and a chat model?

A base model is only pretrained, so it continues text. A chat model is the same base model after extra training on instructions and human feedback, so it answers questions and follows requests.

Why does a base model not follow instructions?

It was only trained to predict the next token of ordinary text. On the web, a question is often followed by more questions, so that is what it writes.

What is few-shot prompting?

Putting a few worked examples in the prompt so the model continues the pattern. It works well on base models because they are excellent at copying patterns.

Why do labs release base models?

Researchers and developers use them as a starting point to fine-tune their own specialised models, or to study what pretraining alone produces.

Related terms