Cross-Validation · Research Atlas

The Concept

Your random forest gets 94% accuracy on the training data. Your supervisor is unimpressed. 'Show me the accuracy on data the model has never seen.' You pull up the model's predictions on the full 847-parcel dataset, restricting to the parcels that weren't used for training, and the number drops to 81%. Still good, but different enough to matter for real decisions.

She explains the trap: any model evaluated on the same data it was trained on will look better than it really is. The model has seen those examples, it's like giving students the exam questions in advance and reporting their test score as a measure of how much they learned.

Cross-validation solves this by repeatedly splitting the data into training and testing sets, in 5-fold cross-validation, the data is split into 5 equal parts. The model is trained on 4 parts and evaluated on the 1 held-out part, then the process repeats for each part in turn. You get 5 independent estimates of real-world accuracy and can average them.

The spread of those 5 estimates tells you something important too: if they range from 62% to 95%, the model's performance is unstable, it's very sensitive to which specific parcels happen to be in the training set. A stable model shows much tighter spread.

The Analogy

Cross-validation is like giving a student five separate pop quizzes on material they have never seen, instead of one exam they got to preview. Some students get lucky on one quiz and unlucky on another, but averaging all five gives a much more honest picture of what they actually know.

The Math (optional)

In k-fold cross-validation, each of the k folds takes a turn as the held-out test set:

k-fold CV score

\text{CV}(k) = \frac{1}{k} \sum_{i=1}^{k} \text{Score}(f_{-i}, \mathcal{D}_i)

f_{-i} is the model trained on all folds except i; D_i is fold i used as the test set. Score(·) is whatever metric matters, accuracy, AUC, RMSE. We average k independent estimates.

Code (optional)

cross_validation.pypython

from sklearn.model_selection import StratifiedKFold, cross_validate
from sklearn.metrics import make_scorer, f1_score

# Stratified k-fold: each fold has the same flood/non-flood ratio as the full data
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

results = cross_validate(
    rf, X, y,
    cv=cv,
    scoring={
        "accuracy": "accuracy",
        "f1": make_scorer(f1_score),
    },
    return_train_score=True,
)

print("Test accuracy:  ", results["test_accuracy"].round(3))
print("Train accuracy: ", results["train_accuracy"].round(3))
# Large gap between train and test = overfitting

Line by line

StratifiedKFold ensures every fold has the same proportion of flooded parcels, important when one class is rarer than the other.
cross_validate returns both train and test scores. A large train-test gap is the diagnostic fingerprint of overfitting.
F1 score is the harmonic mean of precision and recall, more informative than accuracy when flooded parcels are rarer than non-flooded ones.
Printing all 5 test scores (not just the mean) lets you see stability, are some folds much worse than others?

Why Real Researchers Care

Cross-validation is now mandatory in most environmental ML publications. A model reported only with training accuracy is considered unreliable by reviewers, just as a self-marking exam would be considered invalid. Spatial cross-validation (where test and training folds are geographically separated) is increasingly required for spatial prediction tasks to prevent optimistic estimates from nearby data points leaking information.

Quick Check

Q1. Why is evaluating a model on its training data misleading?

Q2. What does a large gap between training accuracy and cross-validated test accuracy indicate?

Your Goal

Given a Bluewater Basin random forest with 98% training accuracy and 76% 5-fold CV accuracy, what would you recommend, and what would you investigate first?

Hint: The gap is large. Think about max_depth, n_estimators, and whether more data would help versus whether the features are genuinely predictive.

Teach It Back

Explain cross-validation to a non-technical colleague using an exam analogy, without using the words 'training', 'testing', or 'fold'.