Skip to content
EN

Lesson 6 of 8 ยท 6 min ยท +100 XP

How does bias get into AI?

AI bias is when an AI system gives worse or unfair results for some groups of people. It usually comes from the training data: some groups are missing or rare, past decisions were already unfair, or labels reflect people's assumptions. The model learns those patterns as if they were rules. Diverse data, testing per group and human review reduce it.

โ–ถ Try it

Why it matters

A model has no opinions, but it copies whatever its data shows. If some people are missing from the examples, the AI works worse for them, without anyone intending it.

In 6 minutes you'll be able to

  • AI bias usually comes from missing, skewed or unfair training data.
  • A system can be biased even if it never sees a sensitive field.
  • Test results per group and keep a human review for important decisions.

What you'll practise

  1. Train the hand sensor

    Most photos in the pile show the same kind of hand. Pick up to 5 training photos, press Train and see who the sensor recognises.

    Hand ยท No hand

    Training examples
    • Light-skinned adult hand, palm up โ†’ Hand
    • Light-skinned hand, fingers spread โ†’ Hand
    • Light-skinned hand, side view โ†’ Hand
    • Another light-skinned hand, waving โ†’ Hand
    • Dark-skinned adult hand โ†’ Hand
    • A child's small hand โ†’ Hand
    • Hand in a work glove โ†’ Hand
    • Empty sink โ†’ No hand
    • A sponge under the tap โ†’ No hand
    • A mug under the tap โ†’ No hand
    Test items
    • A hand with dark skin โ†’ Hand
      Only light-skinned hands were in training, so the sensor learned โ€œhandโ€ means light skin. Nobody intended that; the data did it.
    • A child's hand โ†’ Hand
      No children in the training set, so small hands weren't recognised. Missing groups become groups the AI fails.
    • A hand in a glove โ†’ Hand
      Gloved hands look different. Real users include cleaners and nurses who wear gloves all day.
    • A hand with light skin โ†’ Hand
      Without any light-skinned examples, the sensor had no reference for this hand either.
    • A plate in the sink โ†’ No hand
      The model never saw an object that isn't a hand, so it couldn't tell the difference and switched the tap on.

  2. Ways bias gets in

    Tap each card to see how it happens.

    ๐Ÿ•ณ๏ธ Missing people
    Some groups are rare or absent in the data, so the model works worse for them. Like the tap that didn't see some hands.
    ๐Ÿ“œ Unfair history
    If past decisions were unfair, a model trained on them learns to repeat them, and can make them look objective.
    ๐Ÿท๏ธ Biased labels
    Labels come from people. If labellers share an assumption, the model learns it as a fact.
    ๐Ÿ“ฎ Hidden stand-ins
    Even without asking about a group, a model can pick up clues that go with it, like a postcode or a hobby.

  3. Trace the bias

    1. A company trains a CV-screening AI on ten years of its own hiring, when it mostly hired men. What's the risk?

      • It learns to prefer CVs that look like past hires (Answer)
      • No risk, computers can't be sexist

    2. A model never sees anyone's ethnicity, but uses home postcodes. Can it still be biased?

      • Yes, postcodes can act as a stand-in for groups (Answer)
      • No, removing the field removes the bias

  4. What reduces bias?

    Choose everything that helps.

    • Ask who is missing from the training data
    • Test accuracy separately for different groups
    • Let people affected by decisions ask for a human review
    • Include diverse people when building and testing
    • Trust the result because a computer made it
    • Just add more of the data you already have

Remember

AI copies its data, gaps included. Ask who is missing and test for everyone.

Key facts

  • A 2018 study of three commercial gender-classification systems found error rates of up to 34.7% for darker-skinned women, compared with at most 0.8% for lighter-skinned men. ยท Source: Buolamwini & Gebru, Gender Shades, Conference on Fairness, Accountability and Transparency, 2018
  • Testing nearly 200 face recognition algorithms, the US standards agency NIST found that false positive rates often differed by a factor of 10 to 100 between demographic groups, depending on the algorithm. ยท Source: NIST, Face Recognition Vendor Test Part 3: Demographic Effects (NISTIR 8280), 2019
  • The EU AI Act classifies AI used to recruit or select job candidates as high-risk, which brings requirements such as data quality checks and human oversight. ยท Source: EU Artificial Intelligence Act, Regulation (EU) 2024/1689, Annex III
  • An AI system can look accurate overall while failing a small group badly, because a high average score can hide poor results for the people who are rare in the data.

Updated:

Questions people ask

What is an example of AI bias?

Face analysis systems that make far more mistakes for darker-skinned women than for lighter-skinned men, or a hiring tool trained on past hires that scores some applicants lower because they don't resemble earlier staff.

Can AI be biased if it is just maths?

Yes. The maths learns from data, and data reflects the world and past human decisions. If the examples are skewed or incomplete, the results will be too.

How can AI bias be reduced?

Use diverse and representative data, test results separately for different groups, involve affected people, document limits and keep a human review for important decisions.

What is algorithmic bias?

Algorithmic bias means an automated system produces results that are systematically unfair to certain people or groups. AI bias is the most common form today.

Related terms