Scientific Statistics · Lesson 4 of 7

Statistical Power

~14 min

The Concept

Your test came back non-significant: p = 0.18. Your supervisor doesn't say 'good, there's no change.' She says: 'Before we report this, we need to know if we even had enough power to detect a real change if it existed.'

Statistical power (the probability your test would actually catch a real change if one truly exists) is the probability of detecting a real effect when it's there. A study with low power might miss a genuine trend, not because the trend isn't real, but because the sample was too small or too noisy to see it. Failing to reject H0 is not the same as proving H0 is true.

Power depends on three things: sample size (bigger is more powerful), effect size (larger effects are easier to detect), and significance threshold (stricter α requires more power). You can choose your sample size before collecting data to achieve a desired power, this is called a power analysis.

The Analogy

Imagine a metal detector with a dead battery. Walking over buried coins and finding nothing doesn't prove there are no coins, your detector was too weak to find them. Statistical power is the sensitivity of your test. A high-powered study that finds nothing is meaningful; a low-powered study that finds nothing is ambiguous.

The Math (optional)

Power is computed from the effect size δ (the true difference you care about detecting), standard deviation σ, sample size n, and significance level α.

Required sample size for target power

n \geq \frac{(z_{\alpha/2} + z_{\beta})^2 \cdot 2\sigma^2}{\delta^2}

z_{α/2} is the normal quantile for your significance level (1.96 for α=0.05); z_{β} is for your desired power (0.842 for 80% power); σ is standard deviation; δ is the smallest difference you want to reliably detect.

Code (optional)

power_analysis.pypython

from scipy import stats
import numpy as np

# Parameters for Bluewater Basin nitrate trend detection
sigma = 2.1        # Estimated std dev (mg/L)
delta = 1.5        # Minimum meaningful change to detect (mg/L)
alpha = 0.05       # Significance threshold
power_target = 0.8 # 80% power desired

z_alpha = stats.norm.ppf(1 - alpha/2)
z_beta  = stats.norm.ppf(power_target)

n_required = np.ceil(((z_alpha + z_beta)**2 * 2 * sigma**2) / delta**2)
print(f"Samples per group needed: {int(n_required)}")

# Check actual power with our real sample size
n_actual = 52
power_actual = stats.norm.cdf(
    np.sqrt(n_actual / 2) * delta / sigma - z_alpha
)
print(f"Actual power with n={n_actual}: {power_actual:.3f}")

Line by line

z_alpha and z_beta are quantiles of the normal distribution corresponding to our α and 1−power thresholds.
The formula gives the minimum n per group for two independent samples.
We then check retrospectively: with our actual n, what power did we actually have?

Why Real Researchers Care

Power analysis is required upfront by many funding agencies and journals before a study begins, you must justify that your proposed sample size is adequate to detect the effect you care about. Retrospective power analysis (computing power after a non-significant result) is now standard practice for environmental monitoring, particularly when reporting 'no significant change' findings.

Quick Check

Q1. A study finds p = 0.18 (not significant) with statistical power of only 25%. What is the best interpretation?

Your Goal

Using the formula above (α=0.05, power=0.80, σ=2.1 mg/L), calculate the minimum sample size per group to detect a real nitrate change of 1.0 mg/L at GW-14. How does this compare to Bluewater Basin's current 365-reading/year plan?

Hint: z_{α/2} = 1.96, z_{β} = 0.842. Plug into the formula: n ≥ (1.96 + 0.842)² × 2 × 2.1² / 1.0²

Teach It Back

Explain the difference between 'the test found no significant change' and 'there is no change', using the metal detector analogy.

Previous lesson