Data hygiene and metadata

Why this matters

Most analyses that go wrong go wrong before the first filter: a channel that was never connected, a header that claims the wrong sampling rate, an event table with overlapping timestamps, a “subject” that is two people or one person twice. None of these is subtle once you look, and all of them are invisible if you do not. This lesson turns the looking into a checklist you run on every file, a naming convention that makes files self-describing, and a log that records what you found and decided. Level 0’s capstone is that checklist applied to real files and written up.

Concepts

The first-look checklist

Run this on every recording before anything else, and write down the answers. Each item has a “what wrong looks like”.

1. Sampling rate. Read info['sfreq'], then verify it: the number of samples divided by the recording duration from an independent source (the file’s timestamps, the event spacing, a known block length). A header that says 500 Hz for data recorded at 250 Hz makes every frequency wrong by a factor of two and every latency wrong by half; it is one of the two planted problems in this lesson’s exercise file.
2. Channel count and types. How many channels, how many of each type (EEG, EOG, ECG, trigger, misc), and do the labels match the types? An EOG channel typed as EEG will pollute an average reference; an EEG channel typed as misc will silently vanish from every analysis.
3. Duration. Does it match the protocol (a one-minute baseline, a three-minute block, sixteen alternating blocks)? A file that is too short lost data; a file that is too long may contain a second run, a restart, or a break.
4. Event counts. Count events per code and compare with the design: the expected number of trials per condition, the expected first and last onsets, plausible inter-event intervals. Overlapping or non-monotonic timestamps mean the event stream is corrupt; a code that appears once means a typo or a crash.
5. Power spectral density. One PSD per channel, up to the Nyquist frequency: the alpha peak (if eyes were closed), the mains line and harmonics, the hardware roll-off, the noise floor. A channel whose spectrum is flat and white is not EEG; a channel with a huge low-frequency rise is drifting or disconnected.
6. Flat channels. Zero (or near-zero) variance over the file. One flat channel is usually the recording reference (ds-iowapd’s Pz) or a dead contact; several flat channels are a disconnected connector.
7. DC offsets. A large constant offset on a channel (millivolts, in a DC-coupled recording) means a poor contact or a missing high-pass, and a channel whose offset jumps during the recording had a pop.

Everything here is cheap. The notebook wraps it in a reusable first_look_report() that produces an HTML summary, so that the cost of looking is zero after the first time.

Recognizing flat, DC-offset and mislabelled channels

• Flat: variance at or near zero, or a straight line with only line noise on it. Decide whether it is the reference (expected, keep the knowledge, drop the channel from the montage) or dead (mark bad).
• DC offset: the mean is far from zero and the trace sits off its slot in the viewer; the spectrum has all its power at the lowest frequencies. Not an error in itself in a DC recording, but a warning about the contact.
• Mislabelled: the data contradict the label. A channel called Fp1 that shows no blinks while O1 shows large rounded frontal-looking deflections has had its labels swapped or its cable misrouted; a channel labelled as EEG whose signal is a clean square wave is a trigger line; an “EEG” channel that carries the heartbeat at millivolt scale is the ECG. The check is the topography of things whose topography you know from L0.5: blinks are frontal-polar, alpha is posterior, mu is central. This is the other planted problem in the exercise file.

EEG-BIDS

The Brain Imaging Data Structure extension for EEG (EEG-BIDS, (Pernet et al., 2019) ) fixes where files go and what they are called, so that a dataset describes itself. The pieces you will read and write:

• Folder and file names encode the entities: sub-01/ses-01/eeg/sub-01_ses-01_task-rest_eeg.edf, with run-, acq- and other entities where needed. ds-dortmund uses acq-pre and acq-post for the blocks recorded before and after a cognitive battery within one session, which is not the same as its two longitudinal sessions (ses-1, ses-2): a metadata-reading trap the catalog documents.
• _eeg.json sidecar: SamplingFrequency, PowerLineFrequency, EEGReference, EEGGround, SoftwareFilters, HardwareFilters, RecordingDuration, EEGChannelCount, and the task description. Fill it from the amplifier’s settings, not from the converter’s defaults.
• _channels.tsv: one row per channel with name, type, units, status (good or bad) and status_description. This is where a QC decision about a channel belongs.
• _events.tsv: onset, duration, trial_type, and any columns you need, with _events.json describing them.
• participants.tsv and dataset_description.json: who, and what the dataset is, including its license.

Adopting the convention costs an afternoon once and saves it on every subsequent project; more importantly, every tool that reads BIDS (MNE-BIDS, the BIDS validator, pipeline frameworks in L7.8) can then check your files for you.

Provenance and the QC log

A QC log is a plain record, per file, of what the checklist found and what you decided. Minimum content: file identity (dataset, subject, session, run, a checksum), the loader and its version, the header values and the verified values where they differ, channels marked bad and why, segments marked bad and why, subjects excluded and why, and the date. Put decisions where tools can read them (status and status_description in _channels.tsv, annotations in the raw file) and keep the reasoning in the log. The log is what lets you, or a reviewer, answer “why was S088 excluded?” a year later without re-doing the work.

Provenance is the same idea for derived data: every figure and every number should be traceable to the file, the code and the parameters that produced it. On this site, every shipped snippet carries its dataset, subject, run, start time, license and generating script in a sidecar and in data/manifest.json; your own outputs deserve the same.

Real files with real problems

The catalog documents several data-hygiene cases you will meet on this site:

• ds-eegbci: subjects S088, S089, S092 and S100 carry inconsistent event timestamps (community-reported; not stated on the PhysioNet page), and S038 and S104 are also often dropped. The event-count check catches the timestamp defect, and in S088’s task run R03 the EDF header also reports 128 Hz where every other run reports 160 Hz, so the sampling-rate check fires too (the dropped-samples pitfall shows both against S001). helpers.load_spine excludes the four by default and says so.
• ds-arithmetic: PhysioNet lists N = 36 participants, not the 41 that the file prefixes suggest; counting subjects from file names is a mistake here. ICA was applied upstream, so removed components are unrecoverable.
• ds-dortmund: pre and post are within-session labels, not the two longitudinal sessions.
• ds-lemon: subject counts drift between snapshots of the release; report the N you actually loaded.

Explore

This lesson has no widget. The exploration is the checklist itself, run in the notebook on two files. What to look for, in order:

• the header sampling rate against the rate you compute from samples and duration;
• channel labels against channel content (blinks frontal-polar, alpha posterior, a square wave is a trigger line);
• the event table’s counts, first and last onsets, and whether onsets are monotonic and non-overlapping;
• the per-channel PSD for the alpha peak, the mains line and the hardware roll-off;
• variance per channel, for flat channels and channels with a DC offset that is far from its neighbours.

Practice

The notebook builds first_look_report(), a reusable function that prints and renders as HTML every item of the checklist, then runs it on the two exercise files: a labelled synthetic derivative of ds-eegbci S001R01 with two planted problems (§4.5: synthetic: true, derived_from, modifications), and the real event table of ds-eegbci S088 or S089.

Exercises