Cleaning Scientific Data · Lesson 5 of 6
Documenting Cleaning Decisions
~12 min
The Concept
Six months after you finish cleaning Bluewater Basin's dataset, a new researcher joins the team and asks to replicate your nitrate trend analysis. She opens your cleaned data file. She can see the numbers but has no way of knowing: which outliers you removed and why, which gaps you interpolated vs left as NaN, which calibration corrections you applied, or what the raw data looked like before cleaning.
Without documentation, your cleaned dataset is scientifically incomplete, not wrong, just opaque. Another researcher cannot reproduce your work, check your decisions, or build on it with confidence.
Every cleaning decision must be documented in a data log: what you changed, why, when, and what the original value was. The best practice is to never modify the raw data file, instead, write a cleaning script that reads the raw file and produces a cleaned file, so anyone can re-run the script and get the same result.
This is what 'reproducible research' means at the data level: the path from raw sensor output to analysis-ready data is fully traceable. Anyone with the raw file and the cleaning script can arrive at exactly the same cleaned dataset.
The Analogy
A cleaning log is like a recipe card taped to a dish, instead of just leaving a finished casserole on the counter. Anyone can taste the casserole. Only the recipe card lets someone else make it again, or spot the exact step where too much salt got added.
Code (optional)
"""
Bluewater Basin – WQ-09 dissolved oxygen cleaning pipeline
Author: Your Name
Date: 2025-03-01
Raw input: data/raw/WQ09_DO_raw.csv
Clean output: data/clean/WQ09_DO_clean.csv
Change log: docs/WQ09_cleaning_log.md
"""
import pandas as pd
import numpy as np
RAW = "data/raw/WQ09_DO_raw.csv"
CLEAN = "data/clean/WQ09_DO_clean.csv"
df = pd.read_csv(RAW, parse_dates=["timestamp"])
df = df.set_index("timestamp").sort_index()
changes = []
# Step 1: Replace sentinel values
sentinel_count = (df["do_mgL"] == -9999).sum()
df["do_mgL"] = df["do_mgL"].replace(-9999, np.nan)
changes.append(f"Replaced {sentinel_count} sentinel (-9999) values with NaN")
# Step 2: Flag IQR outliers (do not remove, flag only)
Q1, Q3 = df["do_mgL"].quantile([0.25, 0.75])
IQR = Q3 - Q1
df["flag"] = "OK"
outlier_mask = (df["do_mgL"] < Q1 - 1.5*IQR) | (df["do_mgL"] > Q3 + 1.5*IQR)
df.loc[outlier_mask, "flag"] = "OUTLIER_IQR"
changes.append(f"Flagged {outlier_mask.sum()} IQR outliers (not removed)")
# Step 3: Interpolate short gaps (≤ 1 hour = 4 records at 15-min)
before = df["do_mgL"].isna().sum()
df["do_mgL"] = df["do_mgL"].interpolate(method="time", limit=4)
filled = before - df["do_mgL"].isna().sum()
changes.append(f"Interpolated {filled} short gaps (≤1 hr); {df['do_mgL'].isna().sum()} remain NaN")
df.to_csv(CLEAN)
for line in changes:
print("•", line)Line by line
- The docstring at the top is part of the documentation: it names the input, output, author, date, and where the change log lives.
- We accumulate a changes list as we go, each step appends a plain-English description of what was done.
- Sentinel replacement, outlier flagging, and interpolation are each separate documented steps rather than one combined operation.
- Saving the clean file is the last step, the raw file is never touched, so the cleaning is always reproducible.
Why Real Researchers Care
The FAIR data principles, Findable, Accessible, Interoperable, Reusable, now govern data publication at most journals and funding agencies. The 'R' in FAIR specifically requires that data is accompanied by sufficient documentation for someone else to understand and reuse it. A cleaning script and a change log are the minimum required to meet FAIR standards for a processed dataset.
Quick Check
Q1. Why should you never modify the original raw data file during cleaning?
Your Goal
Write a short data cleaning log entry for one decision from an earlier lesson in this mission, the calibration correction, an outlier decision, or a gap-fill. Include: what you changed, the original value, why you changed it, and the date.
Hint: Use the format: 'Date: [date] | Step: [step name] | Action: [what] | Original: [value or count] | Reason: [why]'
Teach It Back
Explain the concept of a 'reproducible cleaning pipeline' to a researcher who currently edits their raw data files directly in Excel.