Skip to content
EN

Lesson 1 of 6 · 6 min · +100 XP

What is instruction tuning in large language models?

Instruction tuning, also called supervised fine-tuning, is the step that turns a base model into an assistant. The model keeps training on thousands of examples written by people, each a request paired with an ideal answer. It learns the pattern “request, then helpful reply”, so instead of continuing your text, it answers it.

▶ Try it

Why it matters

A freshly pretrained model just continues text: ask it a question and it may write five more questions. A few thousand good examples teach it the shape of a helpful reply.

In 6 minutes you'll be able to

  • A base model continues text. It doesn't know it should answer you.
  • Instruction tuning trains it further on requests paired with ideal answers.
  • The model copies its examples closely, so their quality, variety and tone matter most.

What you'll practise

  1. Autocomplete or assistant?

    The request was: “Give me 3 tips for sleeping better.” Who wrote each reply, the base model or the tuned assistant?

    • Give me 3 tips for eating better. Give me 3 tips for moving more. → Base model
    • 1. Keep the same bedtime. 2. Dim screens an hour before. 3. Skip late coffee. → Tuned assistant
    • …was the most-read post on our forum last March. Reply #1: → Base model
    • Sure! Try a cool, dark room, a fixed wake-up time and no phone in bed. → Tuned assistant
    • Happy to help. Is it falling asleep or staying asleep that's hard? → Tuned assistant

  2. Watch the odds shift

    Be Vireo and pick the next word, before and after tuning. The percentages are illustrative, made up to show the idea.

    1. [Base model, before tuning]
      Name three fruits.       ___

      • Name: 34% (most likely)
      • The: 12%
      • Apple: 8%
      • Bananas: 4%
      • other words: 42%
      Before tuning, the likeliest continuation is another exercise line, like “Name three vegetables.” Lists of questions are common on the web.

    2. [Vireo, after tuning]
      User: Name three fruits.
      Vireo:       ___

      • Apple: 55% (most likely)
      • Sure: 25%
      • Name: 1%
      • other words: 19%
      After thousands of request-and-answer examples, starting the answer is by far the likeliest next word. The list-of-questions habit has almost vanished.

    3. [Vireo, after tuning]
      User: Say “good morning” in Spanish.
      Vireo:       ___

      • Buenos: 70% (most likely)
      • Sure: 15%
      • Good: 5%
      • Say: 1%
      • other words: 9%
      The examples covered many kinds of tasks: lists, translations, summaries. So Vireo follows new requests it never saw word for word.

  3. Choose the training examples

    Tomás is building Vireo's tuning set. Which examples should go in? Pick every good one.

    • Summarise this email in one line → a clear one-line summary
    • What's 15% of 80? → “12”, with the working shown
    • Write a birthday poem for my gran → a warm, short poem
    • Help me get into my ex's account → a kind, clear no
    • What's the capital of Canada? → “Toronto”
    • A random web page, with no request at all
    • Explain rain to a child → 900 words of jargon

Remember

Instruction tuning shows a base model thousands of request-and-answer examples, so answering becomes its most likely next move.

Instruction tuning shows the model request-and-answer examples until answering becomes its most likely next move.

Key facts

  • In the InstructGPT study, people preferred answers from a 1.3-billion-parameter model fine-tuned to follow instructions over those of the 175-billion-parameter GPT-3, a model more than 100 times larger. · Source: Ouyang et al., Training language models to follow instructions with human feedback, OpenAI, 2022
  • Fine-tuning a model on many different tasks written as instructions makes it better at new tasks it was never tuned on. · Source: Wei et al., Finetuned Language Models Are Zero-Shot Learners (FLAN), Google, 2021
  • The LIMA study tuned a large model on just 1,000 carefully chosen examples and still got strong assistant behaviour, suggesting quality matters more than quantity. · Source: Zhou et al., LIMA: Less Is More for Alignment, Meta, 2023
  • Instruction tuning is often called supervised fine-tuning (SFT), because every training example comes with the answer the model should give.

Updated:

Questions people ask

What is the difference between a base model and an instruction-tuned model?

A base model has only learned to continue text, so it may answer a question with more questions. An instruction-tuned model has been trained further on request-and-answer examples, so it responds to what you ask.

What is supervised fine-tuning (SFT)?

Supervised fine-tuning is extra training on examples that each include the correct output. For chat models, those examples are requests paired with ideal answers written or checked by people.

How much data does instruction tuning need?

Far less than pretraining: thousands to hundreds of thousands of examples, compared with trillions of words. Research such as LIMA suggests a small set of excellent examples can go a long way.

Who writes instruction-tuning data?

Usually trained writers and subject experts hired by AI labs, sometimes helped by other models. Their examples set the assistant's style, so labs give them detailed guidelines.

Related terms