Scientific Statistics · Lesson 7 of 7

Effect Size & Practical Significance

~11 min

The Concept

Your test is significant at p = 0.0001. The policymaker is alarmed. But before recommending action, your supervisor asks: 'Significant compared to what? How large is the actual change?' She pulls up the two means: Year 1 was 4.22 mg/L; Year 5 is 4.31 mg/L. The change is 0.09 mg/L, detectable with a large sample, but tiny relative to the WHO limit of 11.3 mg/L.

Statistical significance tells you the effect is real. Effect size tells you whether it matters.

A p-value measures how confident you are that an effect is non-zero. An effect size (a number that says how big the difference actually is, not just whether it exists) measures how large it is. With enough data, even a negligibly small effect will produce a tiny p-value, which is statistically significant but practically irrelevant.

Cohen's d is the most common standardised effect size for comparing two means: it expresses the difference between means in units of standard deviation. A d of 0.2 is 'small', 0.5 is 'medium', 0.8 is 'large'.

The Analogy

Imagine two students both technically pass a spelling test, but one scores 60% and the other scores 99%. Both 'passed', the equivalent of a significant p-value, but the size of the difference tells a completely different story. Effect size is what tells you whether a result is a big deal or barely worth mentioning.

The Math (optional)

Cohen's d standardises the raw difference between means by the pooled within-group standard deviation.

Cohen's d

d = \frac{\bar{x}_2 - \bar{x}_1}{s_p}, \quad s_p = \sqrt{\frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1 + n_2 - 2}}

s_p is the pooled standard deviation, a weighted average of the two groups' variabilities. d = 0 means no difference; d = 1.0 means the means are one full pooled-SD apart.

Code (optional)

effect_size.pypython

import numpy as np

year1 = np.load("gw14_nitrate_year1.npy")
year5 = np.load("gw14_nitrate_year5.npy")

n1, n2 = len(year1), len(year5)
s1, s2 = year1.std(ddof=1), year5.std(ddof=1)

s_pooled = np.sqrt(((n1-1)*s1**2 + (n2-1)*s2**2) / (n1+n2-2))
d = (year5.mean() - year1.mean()) / s_pooled

print(f"Mean difference: {year5.mean()-year1.mean():.3f} mg/L")
print(f"Cohen's d:       {d:.3f}")
print(f"Interpretation: {'small' if abs(d) < 0.5 else 'medium' if abs(d) < 0.8 else 'large'}")

Line by line

s_pooled combines both groups' variance, weighted by their degrees of freedom.
d is dimensionless, it tells you the effect in standard deviation units regardless of the original measurement scale.
The conventional small/medium/large cutoffs are heuristics, not fixed rules, always interpret d in the context of the domain.

Why Real Researchers Care

Since 2019, the American Statistical Association has explicitly discouraged using p < 0.05 as the sole criterion for scientific conclusions, recommending that effect size and uncertainty always accompany significance tests. Nature and Science now require authors to report effect sizes and confidence intervals alongside p-values. The field is shifting toward treating statistical significance as one piece of evidence, not a binary gate.

Quick Check

Q1. A study finds p = 0.0001 with Cohen's d = 0.08. What is the best interpretation?

Your Goal

Calculate Cohen's d for the Year 1 vs Year 5 GW-14 comparison using the values: ȳ₁ = 4.22, ȳ₅ = 8.71, s₁ = 1.8, s₅ = 2.1, n₁ = n₅ = 365. Is this a small, medium, or large effect? Does the magnitude change how you'd advise the policymaker?

Hint: Compute s_pooled first, then d = (8.71 − 4.22) / s_pooled. Compare to Cohen's benchmarks.

Teach It Back

Explain why a very small p-value doesn't automatically mean a large or important effect, using a hypothetical example involving water quality.

Previous lesson