Capstone — Clean pipeline

Brief

Write a script that takes raw BIDS data to cleaned continuous data plus a QC HTML report, per subject, for one full spine dataset — ds-eegbci via its CC0 BIDS mirror (OpenNeuro ds004362), or ds-erpcore. It must run unattended over every subject in the cohort you choose, driven entirely by a configuration file, and it must produce for each subject: a bad-channel table, the ICA components removed with reasons, the percentage of data rejected per condition, the filter settings with their justification, and a run log with package versions and the seed.

This is the level’s synthesis. Every decision you make here was argued somewhere in L2.1–L2.8; the capstone is where they become one artifact that someone else can run.

ds-erpcore is the richer choice because it has two conditions, so the per-condition rejection table is meaningful, and it has EOG channels, so blink handling can be compared across methods. ds-eegbci is the cheaper choice — smaller files, no download barrier, and a task structure simple enough that the pipeline, not the paradigm, stays the object of study. Pick one and say why in the report.

Inputs

• Data. ds-eegbci through its BIDS mirror (ds004362, CC0), or ds-erpcore (CC BY 4.0, open access, per-subject downloadable; 30 EEG + 3 EOG channels, 1024 Hz, CMS reference, 60 Hz mains, no software filters, 40 participants per paradigm). TODO(confirm): the author mirrors the ERP CORE entry into data/registry.yaml (§10.11 item 8) and signs off the dataset page.
• Lessons. L2.1 (loading and BIDS), L2.2 (bad channels and interpolation), L2.3 (re-referencing), L2.4 (filter choices and the two-pass strategy), L2.5 (rejection and its per-condition bias), L2.6 (EOG regression and ICA), L2.7 (ASR and alternatives), L2.8 (order, configuration, logging and QC).
• The documented defects of your dataset. For ds-eegbci: subjects S088, S089, S092 and S100 carry inconsistent event timestamps (S038 and S104 are also commonly dropped), the channel labels need re-casing before a montage will attach, and there are no hardware filters. Whatever you decide about them belongs in the configuration file with its reason, not in a comment.
• The pipelines/ package skeleton from nb-2-8-pipeline, if you built it there: run_subject(config, subject) and the step modules are the same structure this capstone asks for.

Subset rule and runtime

Capstone notebooks run on a documented subset within the §11 limit of ten minutes on Colab’s free tier, and scale to the full cohort locally:

• The documented subset for C2 is 10–20 subjects, listed by ID in the notebook’s first cells. For ds-eegbci the list must state how the documented defective subjects were handled; for ds-erpcore it must state which paradigm (the site’s Level 2 and 3 work uses P3).
• A FULL_COHORT switch at the top of the notebook (default False) extends the run to every subject for local execution. The per-subject function is identical, so the only thing that changes is how many rows the group table has.
• Per-subject products that are expensive to recompute (ICA solutions, per-subject QC numbers) may be precomputed by data/scripts/ and shipped as assets, so the group-level figures can be reproduced without the full download.
• The notebook downloads only the subjects and runs it needs, uses _shared/helpers.py, contains no absolute paths, states every pipeline parameter in one place, and prints in its final cell the numbers the rubric asks for.

Deliverables

1. A configuration file holding every parameter: subject list and exclusions with reasons, montage, filter cutoffs and type, bad-channel criteria and thresholds, reference, ICA method, rank policy and seed, rejection criteria, output paths. Two runs must differ only by this file.
2. run_subject(config, subject) — a function that executes the canonical order (load → montage → bad-channel detection → filter → interpolate → re-reference → ICA → epoch → reject), writes the cleaned continuous data, and returns a structured run log. It must not stop the cohort when one subject fails; a failure is a row in the report.
3. A per-subject QC HTML report containing, at minimum:
   • the bad-channel table: which channels, which criterion fired, in what fraction of windows, which were interpolated, and the resulting rank;
   • the ICA components removed, each with its topography, class, the evidence, and the classifier probability where available, plus the rank after cleaning;
   • the percentage of data rejected per condition, with counts, so condition-biased rejection is visible without asking;
   • the filter settings as resolved (cutoffs, type, length or order, zero-phase or causal) with one sentence per filter saying what it is for;
   • a run log: package versions, the seed, the configuration hash, and the duration of each step;
   • enough figures to see the recording: PSD before and after, ERP or continuous data before and after cleaning, and the data around one flagged segment.
4. A group summary table across subjects: bads interpolated, components removed, percentage rejected per condition, and which subjects the exclusion rule removed and why.
5. A methods paragraph that a reader could follow to reproduce the pipeline, naming every choice and its justification.

Rubric

• Runs unattended on all subjects — one command, no prompts, a failure in one subject does not stop the cohort.
• No condition-biased rejection — the criterion was set condition-blind, and the per-condition table is in the report for every subject.
• QC report readable by a stranger — someone who did not write the pipeline can say what happened to a given subject and whether to trust it.
• Driven entirely by a configuration file, with the seed and the pinned package versions recorded in the run log.
• The canonical order is implemented and each placement is justified in the methods paragraph; any departure from it is argued.
• Rank is tracked through interpolation and re-referencing and passed to ICA (pf-interpolation-rank).
• Every removed ICA component has a recorded class and evidence, and the number removed is comparable across subjects (pf-overcleaning-ica).
• The subject-exclusion rule was fixed in advance and applied blind to any effect.
• Dataset-specific documented defects are handled explicitly — excluded with a reason, or shown to be harmless here.

Example report structure

1. Data and subset. Which dataset, which paradigm or runs, which subjects, why; the FULL_COHORT state of the run being reported.
2. Pipeline. The nine steps in order, each with the one sentence that justifies its placement; the configuration file reproduced or linked.
3. Parameters. Filter settings, bad-channel criteria and thresholds, reference, ICA method and rank policy, rejection criteria, seed — all in one table.
4. Per-subject results. The group summary table; two QC reports shown in full, one typical subject and one problem subject.
5. Quality control. Distribution of bads interpolated, components removed and percentage rejected per condition across subjects; the per-condition imbalance and what you did about it.
6. Exclusions. Who was excluded, by which pre-specified rule, and what the cohort looks like after.
7. Reproducibility. Package versions, seed, configuration hash, runtime; what a reader would have to do to re-run it.
8. Methods paragraph.

Estimated time: TODO(confirm) — the minutes value in the frontmatter is a site-design estimate, not from the spec.