Cleaning Scientific Data · Lesson 3 of 6
Detecting Outliers
~16 min
The Concept
With missing values identified, you turn to the values that are present, and some of them are strange. Well GW-07 shows a nitrate reading of 847 mg/L on a Tuesday afternoon, surrounded by values between 6 and 14. The WHO drinking-water guideline is 50 mg/L. Your first instinct is to delete the 847 immediately.
Your supervisor stops you. 'Is it an instrument error, or did a fertiliser tanker spill upslope that afternoon? You need evidence before you touch it.' She's right. An outlier is not automatically an error, sometimes it's the most important data point in the dataset.
An outlier (a value that sits far away from most of the other values) is a value that sits far from the bulk of the data. It might be an instrument error (sensor malfunction, unit conversion mistake, data entry error), or it might be a genuine extreme event (a pollution spike, a flood, a drought).
The IQR method flags values outside the range [Q1 − 1.5×IQR, Q3 + 1.5×IQR] as potential outliers, where Q1 and Q3 are the 25th and 75th percentiles and IQR = Q3 − Q1. This is robust because it uses the middle 50% of the data to define 'typical', so extreme values don't distort the boundary.
Z-scores flag values more than k standard deviations from the mean. This works well for normally distributed data but can fail when the data is skewed, because extreme values pull the mean toward themselves.
The Analogy
Imagine every kid in class is between 120cm and 150cm tall, except one kid who is 210cm. Do not immediately assume the measuring tape broke. Maybe it did. Or maybe that kid is genuinely, wonderfully tall, and is exactly the interesting fact you were supposed to notice.
The Math (optional)
Two common outlier detection statistics for environmental data:
IQR method bounds
Q1 and Q3 are the 25th and 75th percentiles. IQR = Q3 − Q1. Values outside [lower, upper] are flagged.
Z-score
x̄ is the sample mean, s is the standard deviation. A z-score tells you how many standard deviations a value sits from the mean. |z| > 3 is a common flagging threshold.
Code (optional)
import pandas as pd
import numpy as np
nitrate = pd.read_csv("GW07_nitrate.csv")["nitrate_mgL"]
# IQR method
Q1, Q3 = nitrate.quantile(0.25), nitrate.quantile(0.75)
IQR = Q3 - Q1
lower, upper = Q1 - 1.5 * IQR, Q3 + 1.5 * IQR
iqr_outliers = nitrate[(nitrate < lower) | (nitrate > upper)]
# Z-score method
z = (nitrate - nitrate.mean()) / nitrate.std()
zscore_outliers = nitrate[z.abs() > 3]
print(f"IQR flags: {len(iqr_outliers)} values")
print(f"Z-score flags: {len(zscore_outliers)} values")
print("\nIQR-flagged values:")
print(iqr_outliers)Line by line
- quantile(0.25) and quantile(0.75) give Q1 and Q3 without being influenced by extreme values.
- The IQR bounds are computed; any reading outside them is flagged, not deleted, just flagged for investigation.
- The Z-score method divides each deviation from the mean by the standard deviation; absolute values above 3 are unusual.
- We print the flagged values rather than removing them, the decision about what to do comes after a human investigates.
Why Real Researchers Care
Professional environmental data workflows always flag rather than delete outliers in a first pass. Flagged data is reviewed against field notes, calibration records, and nearby sensors before any removal decision. This workflow, detect, investigate, decide, document, is codified in the USGS Guidelines for Quality Assurance and Quality Control for Environmental Monitoring.
Quick Check
Q1. Well GW-07 shows a nitrate value of 847 mg/L flagged as an outlier. What should you do first?
Your Goal
For Bluewater Basin's nitrate dataset below, compute Q1, Q3, IQR, and the outlier bounds. Then identify which, if any, values are flagged: [4.2, 5.1, 6.8, 7.2, 8.1, 9.3, 10.2, 11.1, 12.8, 847.0]
Hint: Sort the data, find the 25th and 75th percentile positions, compute IQR = Q3 − Q1, then set bounds at Q1 − 1.5×IQR and Q3 + 1.5×IQR.
Teach It Back
Explain why an outlier might be the most important data point in a dataset, using a real-world environmental scenario as your example.