How LLMs Are Made / Glossary

Embedding

An embedding is a list of numbers that represents a word, token, sentence or image as a point in a space of meaning, so that things with similar meanings end up close together.

A model can't compute with words, so it gives each token a list of numbers. Training moves these points so that words used in similar sentences, like cat and kitten, sit near each other and unrelated words sit far apart.

Directions in this space carry meaning too: with word vectors learned from text, king minus man plus woman lands close to queen. In the largest GPT-3 model, every token was turned into a list of 12,288 numbers.

Embeddings are also used outside chatbots, for example in search that finds documents by meaning rather than exact words. Because they are learned from human text, they can pick up stereotypes from that text as well.