The Reality of Raw Data

You open the first raw data file from Bluewater Basin's water quality network. Station WQ-09, twelve months of dissolved oxygen readings at fifteen-minute intervals, 35,040 rows. The first thing you notice is a value of -9999. Then another. Then a whole week where every reading is exactly 0.000. Then a column labelled 'DO_mgL' that contains the text 'MAINTENANCE' in row 14,837.

Welcome to real scientific data. Your supervisor has seen your expression before. 'Every dataset I've ever worked with,' she says, 'looked like this on first open. Cleaning it is not a failure of the fieldwork, it is the fieldwork.'

Raw sensor data is almost always imperfect. Sensors go offline, get clogged with sediment, lose power, get swapped mid-deployment, or simply malfunction for no documented reason. When this happens, data loggers record whatever they can, sometimes a sentinel value like -9999 to flag 'no reading', sometimes zero, sometimes nothing at all.

Data cleaning, or Quality Assurance / Quality Control (QA/QC, meaning the checks that catch and fix bad data before anyone trusts it), is the process of identifying and handling these problems in a principled, documented way. The goal is not to make the data look better; it is to make the data accurately represent what was actually measured, so that later analysis isn't fooled by instrument artefacts.

Raw data is like produce straight from the farm: some of it is perfect, some has bruises, and a few pieces are rotten all the way through. Cleaning is the sorting table before anything goes in the basket. Skipping it does not mean you avoided the rotten piece, it means it ends up in someone's soup.

It's widely estimated that environmental data scientists spend 50–80% of their project time on data cleaning rather than analysis. This is not inefficiency, it is science. A single undetected -9999 value, passed through to a mean calculation, would produce a result orders of magnitude wrong. Professional data quality guidelines (ISO 5667-14, USGS QA/QC manuals) treat cleaning as a core scientific activity, not a preprocessing chore.

Explain to a colleague why data cleaning is a scientific activity rather than a technical chore, using one concrete example from this lesson.