What are embeddings, and how does a model know words are similar?
An embedding is a list of numbers that places a word, or token, as a point in a space of meaning. Words used in similar sentences, like cat and kitten, end up close together, while unrelated words sit far apart. Directions carry meaning too, so king minus man plus woman lands near queen. Models learn these points from text.
▶ Try it
Why it matters
A token ID is just a label. The embedding turns it into a position, so the model can tell that puppy is like dog and nothing like bus. Search and recommendations use the same trick.
In 6 minutes you'll be able to
- When a search finds what you meant rather than what you typed, embeddings are usually at work.
- Words that look alike, like dog and hot dog, can be far apart: meaning comes from use, not spelling.
- Because embeddings learn from human text, ask how a tool checks them for stereotypes.
What you'll practise
Close or far?
On Vireo's map of meaning, are these pairs close together or far apart?
- cat · kitten → Close together
- car · banana → Far apart
- doctor · nurse → Close together
- hot · cold → Close together
- piano · volcano → Far apart
- Paris · Rome → Close together
The meaning map
Drag each word onto the map where you think it belongs. Then try word maths. The map is a simplified 2D picture; real embeddings have thousands of directions.
- puppy → dog, cat. A puppy is a young dog and turns up in the same sentences, so it lands with the animals.
- hot dog → bread, cheese. Position follows meaning, not spelling. A hot dog is food, so it lands by the bread, far from dog.
- bicycle → car, train. Bicycles appear in sentences about roads, riding and traffic, so they sit near the other vehicles.
- cheerful → happy, sad. Cheerful is used almost exactly like happy, so the two are next-door neighbours.
- king − man + woman ≈ queen (man → king). The arrow from man to king is a direction that means roughly “royal”. Taking the same step from woman lands near queen.
- Paris − France + Italy ≈ Rome (country → capital). The step from a country to its capital points the same way for many countries. From Italy, it lands on Rome, not Milan.
Where you meet embeddings
You search a help site for “money back” and it finds the page titled “Refunds”. How?
- Their embeddings are close, even though the words differ (Answer)
- Someone typed every possible synonym by hand
Embeddings learned from old texts link some jobs more to men than to women. What follows?
- Teams need to test for bias and correct it (Answer)
- Nothing, maths can't be biased
Remember
An embedding gives every token an address in a space of meaning. Close means similar, and directions mean relationships.
Every token becomes a point in a space of meaning. Close points mean similar use, and directions mean relationships.
Key facts
- The idea that a word's meaning comes from the words around it was summed up in 1957: “You shall know a word by the company it keeps.” · Source: J. R. Firth, A Synopsis of Linguistic Theory, 1957
- Word vectors learned from text showed that simple arithmetic captures relationships: king − man + woman lands close to queen. · Source: Mikolov et al., Google, Linguistic Regularities in Continuous Space Word Representations, NAACL 2013
- In the largest GPT-3 model, every token was turned into a list of 12,288 numbers. · Source: Brown et al., OpenAI, Language Models are Few-Shot Learners, 2020
- Word embeddings trained on news text were shown to carry gender stereotypes about jobs, and researchers proposed ways to reduce them. · Source: Bolukbasi et al., Man is to Computer Programmer as Woman is to Homemaker?, NeurIPS 2016
Updated:
Questions people ask
What is an embedding in simple terms?
It's a list of numbers that works like an address for a word or a piece of text. Things with similar meanings get nearby addresses, so a computer can measure how alike they are.
How many dimensions does an embedding have?
Far more than the two on a drawing. Large language models use hundreds to many thousands of numbers per token. The 2D map is only a simplified picture.
What are embeddings used for?
Inside a language model they are the first step of reading. Outside, they power search by meaning, recommendations, grouping similar documents and finding duplicates.
Does king minus man plus woman really equal queen?
Roughly. In classic word embeddings the result lands closer to queen than to most other words. It's a famous demonstration that directions in the space carry meaning, though it doesn't work for every example.
Related terms