Machine Learning · Lesson 3 of 5

From One Tree to a Forest

~18 min

The Concept

A single decision tree has a flaw: it's brittle. One different training example and the root split might change entirely, taking every prediction below it with it. Your supervisor demonstrates this by rerunning the tree on a slightly different subset of the survey data, the tree structure shifts noticeably.

'The solution,' she says, 'is the same one used by survey teams, juries, and committees everywhere, ask many independent judges and take the majority vote.' A random forest (a big group of decision trees that each vote, so no single tree's mistake wins) is an ensemble of decision trees, each trained on a different random sample of the data. The individual trees are noisy; the ensemble is stable.

A random forest builds many trees independently, each on a bootstrap sample, a random sample with replacement from the training data, about the same size as the original. Some parcels appear multiple times; others are left out entirely. This means each tree sees a slightly different view of the data and makes different errors.

When you need to predict a new parcel, every tree in the forest votes: flooded or not flooded. The forest predicts whichever class gets more votes. The individual trees' errors tend to cancel out in the majority vote, this is called variance reduction through ensembling.

The Analogy

It is like asking a hundred different friends to independently guess how many jellybeans are in a jar, instead of trusting just one guess. Each individual friend might be way off, but averaging all hundred guesses together usually lands remarkably close to the true number, because their individual mistakes tend to cancel out.

The Math (optional)

Bootstrap sampling and the aggregation of predictions (bagging) are what separate a forest from a single tree:

Bootstrap sample

\mathcal{D}_b \sim \text{Sample with replacement}(\mathcal{D}, n)

Each tree b is trained on a new bootstrap dataset Db drawn by sampling n times with replacement from the original data D. About 63% of original examples appear; the rest are 'out-of-bag'.

Forest prediction

\hat{y} = \text{mode}\{f_1(x), f_2(x), \ldots, f_B(x)\}

The final prediction is the majority vote across all B trees. For regression (predicting a number instead of a class) it's the average instead of the mode.

Try It

Grow the forest tree by tree. Watch feature importance stabilise as more trees vote, and notice that no single tree dominates the decision.

Feature importance across 1 tree

Slope (%)

33%

Upstr. area (km²)

50%

Soil drainage

17%

Each tree trains on a bootstrap sample of 120 Bluewater Basin sites. The forest votes: a site is predicted flooded if the majority of trees say so. Feature importance shows which variables drive the most splits across all trees.

Code (optional)

random_forest.pypython

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
import numpy as np

rf = RandomForestClassifier(n_estimators=100, max_depth=6, random_state=42)
rf.fit(X_train, y_train)

# Cross-validated accuracy (5-fold)
cv_scores = cross_val_score(rf, X, y, cv=5, scoring="accuracy")
print(f"CV accuracy: {cv_scores.mean():.3f} ± {cv_scores.std():.3f}")

# Feature importance
for feat, imp in sorted(zip(features, rf.feature_importances_), key=lambda x: -x[1]):
    print(f"  {feat:30s}: {imp:.3f}")

Line by line

RandomForestClassifier with n_estimators=100 builds 100 trees, each on a different bootstrap sample.
cross_val_score with cv=5 is 5-fold cross-validation: we'll explain this fully in the next lesson. The ± gives the uncertainty of the accuracy estimate.
rf.feature_importances_ shows how much each feature contributed to splits across all trees, a data-driven way to identify which variables actually drive flood risk.

Why Real Researchers Care

Random forests are one of the most widely deployed ML models in environmental science because they handle mixed variable types, are robust to outliers, provide built-in feature importance, and require relatively little tuning compared to neural networks. A 2019 meta-analysis found random forests outperformed all other methods on 68% of environmental prediction benchmarks.

Quick Check

Q1. Why does bootstrap sampling help a random forest?

Your Goal

Grow the forest to 12 trees in the simulation. Note which feature has the highest importance. Does this match what you'd expect physically from a flooding perspective?

Hint: Slope controls how fast water moves; upstream area controls how much water arrives. Both should matter, which matters more?

Teach It Back

Explain why a majority vote of 100 noisy trees is more reliable than one carefully built tree, without using any technical terms.

Previous lesson