Quantifying Uncertainty · Lesson 5 of 6
Calibration
~12 min
The Concept
Your model has been producing '90% prediction intervals' for six months of streamflow forecasts. Your supervisor asks: 'has the true value actually landed inside your interval about 90% of the time, or are we just saying 90% and hoping?'
A model is well-calibrated (its confidence claims actually match how often it turns out to be right) if its stated uncertainty matches its actual error rate: a well-calibrated 90% interval really does contain the truth about 90% of the time, checked retrospectively against real outcomes. A model can have impressively narrow intervals and still be badly calibrated if those narrow intervals are wrong more often than they claim to be.
The Analogy
Think of a weather forecaster who says '90% chance of rain' every single day, rain or shine, regardless of the actual weather. If it only rains on 40% of those days, the forecaster is badly calibrated, their confidence numbers do not match reality, even if they sound precise every time.
Code (optional)
import numpy as np
# For each of 24 past months: was the true streamflow inside the stated 90% interval?
lower_bounds = np.array([...])
upper_bounds = np.array([...])
true_values = np.array([...])
covered = (true_values >= lower_bounds) & (true_values <= upper_bounds)
coverage_rate = covered.mean()
print(f"Stated interval: 90%")
print(f"Actual coverage over 24 months: {coverage_rate * 100:.1f}%")Line by line
- We check, for every historical prediction, whether the true observed value actually fell inside the interval that was claimed at the time.
- Averaging that yes/no coverage across all 24 months gives the empirical coverage rate.
- Comparing 90% (stated) against the empirical rate tells us directly whether the model is honest about its own uncertainty.
Why Real Researchers Care
Calibration checks are standard practice for any operational forecasting system, weather services, disease forecasting dashboards, and financial risk models are all routinely audited this way, because a model that is confidently wrong is often more dangerous than one that admits high uncertainty.
Quick Check
Q1. What does it mean for a model to be 'badly calibrated'?
Your Goal
If a check like the one above found your 90% intervals actually contained the truth only 65% of the time, what would you recommend doing before using this model for any more forecasts?
Hint: Consider whether the intervals need to be widened, or whether the underlying model needs revisiting.
Teach It Back
Explain calibration to someone using a weather forecaster analogy: what would it mean for a forecaster's '70% chance of rain' claims to be well-calibrated?