↓Skip to main content

You are probably bad in probability

Table of Contents

In her book “Randomness,” Deborah Bennett presents an example that illustrates how bad we are at calculating probabilities, even highly trained professionals. A group of doctors received this question:

The test of a disease has a 5% false positive rate. The disease affects 1 in 1000 people in the population. People are tested randomly, whether they are suspected to be sick or not. A particular patient’s test is positive. What is the probability that this patient actually has the disease?

Most doctors said 95%. The correct answer is closer to 2%.

An Intuitive Approach #

Let’s break this down step by step. Imagine you have a machine (the test) to find sick people in a village of 1000 inhabitants:

  1. In this village, only 1 person is actually sick (1 in 1000)
  2. The machine makes mistakes: for every 100 healthy people it tests, it gets it wrong with 5 and says they’re sick when they’re not (5% false positive rate)

Out of the 1000 people in the village:

  • 1 person is actually sick
  • 999 people are healthy

Of the 999 healthy people:

  • The machine gets it wrong with 5% of them
  • 5% of 999 ≈ 50 people
  • This means the machine will say 50 healthy people are sick

So, when the test returns a positive result:

  • It could be the 1 person who is actually sick
  • Or it could be one of the 50 healthy people the machine got wrong about

So only 1 among 51 positives is really sick. Probability ≈ 2%

Why Did the Doctors Get It Wrong? #

The doctors’ intuitive answer of 95% was wrong because they focused only on the test’s accuracy for a single case. They failed to consider:

  1. The base rate (how rare the disease is)
  2. How the false positive rate affects a large population of healthy people
  3. How these factors combine using Bayes’ Theorem

This is known as the base rate fallacy or base rate neglect, and it’s a common cognitive bias that affects all of us.

More #

  • Tversky, A., & Kahneman, D. (1974). “Judgment under Uncertainty: Heuristics and Biases.” Science, 185(4157), 1124-1131. The landmark paper introducing cognitive biases in probability estimation. First formal description of base rate neglect

Note: We are ignoring the false negatives ratio here for clarity.