How LLMs Are Made / Glossary

Base model

A base model is a language model straight out of pretraining. It continues text instead of following instructions, so it needs further training before it works as an assistant.

Ask a base model a question and it may reply with more questions, because on the web a list of questions is often followed by more questions. It has learned to continue documents, not to help the person typing.

Base models still know a lot. GPT-3 showed that a base model can do new tasks from a few examples placed in the prompt, called few-shot prompting, as long as the text looks like a document where the answer comes next.

Many labs release both a base model, for researchers and developers to build on, and an instruction-tuned version for chatting. The later assistant training uses far less data and mainly shapes how the model behaves.