In her book “Randomness,” Deborah Bennett presents an example that illustrates how bad we are at calculating probabilities, even highly trained professionals. A group of doctors received this question:
The test of a disease has a 5% false positive rate. The disease affects 1 in 1000 people in the population. People are tested randomly, whether they are suspected to be sick or not. A particular patient’s test is positive. What is the probability that this patient actually has the disease?
Most doctors said 95%. The correct answer is closer to 2%.
An Intuitive Approach
Let’s break this down step by step. Imagine you have a machine (the test) to find sick people in a village of 1000 inhabitants:
- In this village, only 1 person is actually sick (1 in 1000)
- The machine makes mistakes: for every 100 healthy people it tests, it gets it wrong with 5 and says they’re sick when they’re not (5% false positive rate)
Out of the 1000 people in the village:
- 1 person is actually sick
- 999 people are healthy
Of the 999 healthy people:
- The machine gets it wrong with 5% of them
- 5% of 999 ≈ 50 people
- This means the machine will say 50 healthy people are sick
So, when the test returns a positive result:
- It could be the 1 person who is actually sick
- Or it could be one of the 50 healthy people the machine got wrong about
So only 1 among 51 positives is really sick. Probability ≈ 2%
Why Did the Doctors Get It Wrong?
The doctors’ intuitive answer of 95% was wrong because they focused only on the test’s accuracy for a single case. They failed to consider:
- The base rate (how rare the disease is)
- How the false positive rate affects a large population of healthy people
- How these factors combine using Bayes’ Theorem
This is known as the base rate fallacy or base rate neglect, and it’s a common cognitive bias that affects all of us.
More
- Tversky, A., & Kahneman, D. (1974). “Judgment under Uncertainty: Heuristics and Biases.” Science, 185(4157), 1124-1131. The landmark paper introducing cognitive biases in probability estimation. First formal description of base rate neglect
Note: We are ignoring the false negatives ratio here for clarity.