Simpson’s Paradox can make AI systems look biased or fair depending on how you slice the data. Here’s why it breaks fairness audits — and how to fix it.
When Fairness Flips
Imagine you’re auditing a hiring algorithm for gender fairness. At first glance, the data looks balanced — men and women are hired at roughly the same rate.
But when you break it down by department, you discover that women are consistently hired less often within every single department.
How can both statements be true? Welcome to Simpson’s Paradox — the statistical trap that can make an AI system look both fair and biased, depending on how you slice the data.
💡 The Real-World Case
This isn’t a thought experiment. In the 1970s, UC Berkeley was sued for alleged gender bias in graduate admissions.
- Aggregate view: Men were admitted at 44%, women at 35%. Looked biased.
- Department-level view: Within most departments, women had higher admission rates than men.
The paradox was that women applied disproportionately to competitive departments (with low admission rates for everyone). Aggregated data screamed bias; subgroup data told the opposite story.
