Regression & Prediction · Lesson 4 of 7
Overfitting & Cross-Validation
~16 min
The Concept
A colleague adds ten predictors to the streamflow model, some physically meaningful, some not, and achieves R² = 0.94 on the training data. Impressed, the team prepares to deploy the model for flood forecasting. Your supervisor asks one question before approving it: 'What's its R² on data it hasn't seen?'
The colleague tests it on the next year's data. R² = 0.41. The model has learned the quirks and noise of the training set so thoroughly that it fails completely on new data. This is overfitting.
Overfitting (when a model memorizes the exact data it was shown instead of learning the general pattern) means a model has memorised the training data rather than learned the underlying relationship. It looks brilliant on data it's seen and terrible on data it hasn't. A model with too many parameters relative to the data available will almost always overfit.
Cross-validation (testing a model on data it has never seen, to check if it really learned something useful) is the cure: split your data into a training set and a test set. Fit on training, evaluate on test. The model never sees the test data during fitting, its performance there is a realistic estimate of how well it will generalise to future data.
The Analogy
It is like a student who memorizes the exact answers to last year's practice test instead of learning the underlying material. They will ace that exact practice test, but bomb the real exam, which asks similar questions in a slightly different way. Cross-validation is giving the student a brand new, unseen practice test to find out whether they actually learned anything.
Code (optional)
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score, KFold
df = pd.read_csv("bluewater_monthly_features.csv")
X = df.drop("streamflow", axis=1)
y = df["streamflow"]
model = LinearRegression()
kf = KFold(n_splits=5, shuffle=True, random_state=42)
cv_r2 = cross_val_score(model, X, y, cv=kf, scoring="r2")
print(f"Train R²: {model.fit(X, y).score(X, y):.3f}")
print(f"CV R² per fold: {cv_r2.round(3)}")
print(f"Mean CV R²: {cv_r2.mean():.3f} ± {cv_r2.std():.3f}")Line by line
- KFold divides the data into 5 equal parts ('folds'); in each iteration, 4 folds train the model and 1 fold tests it.
- cross_val_score fits and evaluates on each fold automatically, returning 5 R² values.
- The gap between train R² and mean CV R² reveals overfitting: a large gap means the model learned noise.
Why Real Researchers Care
Cross-validation is now a mandatory step in most environmental modelling workflows. The USGS's National Hydrologic Model framework, NOAA flood prediction systems, and the UK's National Flood Forecasting System all use held-out validation to report honest model performance. Quoting only training R² is considered poor practice in applied hydrology.
Quick Check
Q1. A model has training R² = 0.94 and 5-fold cross-validation R² = 0.62. What does this mean?
Your Goal
Your 4-predictor model has CV R² = 0.78. Adding 4 more predictors raises training R² from 0.82 to 0.91, but CV R² drops to 0.71. What would you report as the model's performance, and which model would you deploy?
Hint: CV R² is the honest estimate of real-world performance. Always choose the simpler model unless CV clearly improves.
Teach It Back
Explain what overfitting is to a scientist who knows nothing about machine learning, using a concrete analogy that doesn't involve statistics.