Scientific Statistics · Lesson 5 of 7
Multiple Testing & False Discovery
~12 min
The Concept
Emboldened by the hypothesis test, a junior team member runs the same test on all fifteen Bluewater Basin wells and reports that two of them show significant increases (p < 0.05). Your supervisor asks: 'If there were truly no change at any well, how many would you expect to come back significant at α = 0.05?'
The answer is 0.75, roughly one well. When you test fifteen hypotheses simultaneously at α = 0.05, you should expect false positives even when nothing is actually changing.
The multiple comparisons problem (the trap of running many tests and mistaking pure luck for a real finding) arises whenever you conduct many tests simultaneously. Each test has a 5% chance of a false positive if the null is true. With 15 tests, the probability of at least one false positive approaches 54%, even if the null is true everywhere.
The Bonferroni correction (a simple way to make each individual test stricter when you are running many tests at once) is the simplest fix: divide your significance threshold by the number of tests. For 15 tests at α = 0.05, require p < 0.05/15 = 0.0033 for any individual test. It's conservative but easy to apply and justify.
The Analogy
If you buy one lottery ticket, winning would be shocking. If you buy a thousand tickets, someone winning is not shocking at all, it is expected. Testing fifteen wells for a trend is like buying fifteen tickets: finding one or two 'wins' by pure chance is exactly what you should expect, not proof that something real happened.
The Math (optional)
The Bonferroni-corrected threshold controls the family-wise error rate, the probability of any false positive across all m tests.
Bonferroni correction
Divide your desired family-wise error rate α by the total number of tests m. Each individual test must achieve this stricter threshold to be considered significant.
Code (optional)
import numpy as np
from scipy import stats
# Simulate 15 hypothesis tests with truly no change (H0 true everywhere)
np.random.seed(42)
p_values = []
for _ in range(15):
y1 = np.random.normal(5.0, 2.1, 52) # Year 1
y2 = np.random.normal(5.0, 2.1, 52) # Year 5 (same distribution)
_, p = stats.ttest_ind(y2, y1, alternative="greater")
p_values.append(p)
uncorrected = sum(p < 0.05 for p in p_values)
bonferroni = sum(p < 0.05/15 for p in p_values)
print(f"Significant at α=0.05 (uncorrected): {uncorrected} / 15")
print(f"Significant at Bonferroni-corrected: {bonferroni} / 15")Line by line
- We simulate the case where H0 is true everywhere, both years are drawn from the exact same distribution.
- uncorrected counts how many tests produce p < 0.05 by chance alone, this is the false positive rate in practice.
- Bonferroni correction applies 0.05/15 ≈ 0.0033 per test, dramatically reducing false positives.
Why Real Researchers Care
Multiple testing corrections are mandatory in fields where many simultaneous tests are routine, genomics (thousands of genes), environmental monitoring (dozens of stations), neuroimaging (thousands of brain voxels). The false discovery rate (FDR), controlled by the Benjamini-Hochberg procedure, is often preferred to Bonferroni for large numbers of tests because it's less conservative while still limiting the expected proportion of false discoveries.
Quick Check
Q1. You test 20 monitoring stations for a trend and find 3 significant at p < 0.05. How many false positives would you expect if no station were actually changing?
Your Goal
After applying Bonferroni correction to 15 wells (α = 0.05), what is the corrected threshold? If GW-14 has p = 0.012, does it survive the correction?
Hint: Corrected α = 0.05 / 15. Compare 0.012 to that threshold.
Teach It Back
Explain to a policymaker who wants to act on any significant result why running fifteen tests and reporting the two that came back positive might be misleading.