nb-c2-clean-pipeline · Capstone C2 — Clean pipeline¶
Capstone C2 · Level 2 · Status draft — for expert review; uncertain points carry TODO(confirm).
Brief (spec §6, C2). A script that takes raw BIDS → cleaned continuous data + a QC HTML report per subject for one full spine dataset, driven entirely by a configuration file, running unattended over every subject, and producing for each: a bad-channel table, the ICA components removed with reasons, the percentage of data rejected per condition, the filter settings with their justification, and a run log with package versions and the seed. Rubric: runs unattended on all subjects; no condition-biased rejection; the QC report is readable by a stranger.
Dataset choice, and why. ds-erpcore P3 rather than ds-eegbci. It has two conditions, so the per-condition rejection table means something and the rubric's central check is testable; it has three EOG channels, so blink handling can be compared across methods (L2.6, L2.7); and it is the paradigm the rest of Level 2 and all of Level 3 use, so the pipeline that comes out of here is the one C3 reuses unchanged. ds-eegbci is the cheaper choice — smaller files, no download barrier — and its BIDS mirror (OpenNeuro ds004362, CC0) is the entry point nb-2-1-bids uses; pipelines/configs/eegbci.yaml runs the identical pipeline on it if you would rather.
Subset rule and runtime (§11). The documented subset is sub-001 … sub-010 — ten of the paradigm's forty participants, listed by ID below, about 560 MB of downloads on a machine with an empty cache. Setting FULL_COHORT = True in the configuration cell runs the identical code over all forty (~2.2 GB of downloads; local runs only). The per-subject function does not change; only the number of rows in the group table does.
What it produces. Per subject: a cleaned continuous recording, the surviving epochs, a machine-readable run log and a self-contained QC HTML page. Across subjects: a cohort index, a group summary table, a condition-bias check, and a methods paragraph. Nothing is written into the repository; the output directory is a temporary folder by default and EEG_COURSE_C2_OUT points it elsewhere.
TODO(confirm): the author mirrors the ERP CORE entry into the catalog registry and signs off the dataset page (§10.11 item 8). The licence is contested at source and three real statements disagree: the LICENSE file shipped with the data says CC BY-SA 4.0, dataset_description.json says CC0, and the OSF node record says CC BY 4.0. Spec §10.7 makes the most restrictive reading govern, so the site records CC BY-SA 4.0 (§13 item 15, answered 2026-09-18). Share-alike binds anything you derive from these data, including whatever this capstone produces.
In [1]:
# Setup: dependencies, the shared helpers, non-interactive plotting.
import importlib.util
import subprocess
import sys
import warnings
from pathlib import Path
# 1. Dependencies are pinned in notebooks/requirements.txt. Nothing is installed
# when the pinned stack is already present (local runs, CI); a fresh Colab or
# Binder kernel installs it once. On Colab, run from a clone of the repository
# so that notebooks/_shared/ is available (repository URL: TODO(confirm), spec
# section 13 item 3).
_needed = ('mne', 'scipy', 'matplotlib', 'pooch', 'pyprep', 'autoreject', 'mne_bids', 'yaml')
_missing = [p for p in _needed if importlib.util.find_spec(p) is None]
if _missing:
_req = next((d / "requirements.txt" for d in (Path.cwd(), *Path.cwd().parents)
if (d / "requirements.txt").exists()), None)
_cmd = [sys.executable, "-m", "pip", "install", "-q"]
_cmd += ["-r", str(_req)] if _req else ["mne==1.10.2", "pooch>=1.8"]
if "pyprep" in _missing:
_cmd += ["pyprep>=0.9"]
if "autoreject" in _missing:
_cmd += ["autoreject>=0.5"]
if "mne_bids" in _missing:
_cmd += ["mne-bids>=0.16"]
if "yaml" in _missing:
_cmd += ["pyyaml>=6"]
subprocess.check_call(_cmd)
# 2. Shared helpers (notebooks/_shared/helpers.py and helpers_l2.py), located
# relative to the working directory -- notebooks/<level>/ or notebooks/ --
# never through an absolute path.
_shared = next((d / "_shared" for d in (Path.cwd(), *Path.cwd().parents)
if (d / "_shared" / "helpers_l2.py").exists()), None)
if _shared is None:
raise FileNotFoundError("start the kernel in notebooks/L2/ (or notebooks/) so that _shared/helpers_l2.py is found")
sys.path.insert(0, str(_shared))
import helpers
import helpers_l2 as l2
# 3. Plotting: Jupyter's default inline backend renders static PNGs through Agg
# (no windows, nothing blocks); outside Jupyter the helpers select Agg. Every
# MNE figure is requested with show=False, and plt.show() renders each cell's
# figures in place.
import matplotlib.pyplot as plt
import numpy as np
import mne
# Warnings are worth reading, so they are not silenced -- but their default format prints the
# absolute path of the file that raised them, which is nobody else's business and would put this
# machine's directory layout into the saved outputs. Only the class and the message are shown.
warnings.formatwarning = lambda message, category, *a, **k: f"{category.__name__}: {message}\n"
mne.viz.set_browser_backend("matplotlib", verbose=False)
mne.set_log_level("WARNING")
plt.rcParams["figure.dpi"] = 72
print(f"MNE {mne.__version__}; helpers imported from notebooks/_shared")
print("ERP CORE downloads (~56 MB per subject) are cached under data/downloads/erpcore/ in a repository "
"clone, otherwise under MNE's data directory; nothing is re-fetched.")
1. The pipeline package, and the configuration that drives it¶
Every parameter of the pipeline — the subject list, the exclusions with their reasons, the montage, the filter cutoffs and their written justification, the detection thresholds, the reference, the ICA method and rank policy, the rejection criteria, the seed — lives in one YAML file. Nothing is typed at a prompt; nothing is commented out to switch behaviour. The two overrides below are arguments, not edits: where the output goes, and (because mne-icalabel is installed here) that ICLabel should give its second opinion on every component.
FULL_COHORT is the §11 switch: it changes the subject list and nothing else.
In [2]:
import json
import os
import shutil
import tempfile
import time
from IPython.display import HTML, display
FULL_COHORT = False # True: all 40 participants (~2.2 GB of downloads; local runs only)
_pipelines = next((d / "pipelines" for d in (Path.cwd(), *Path.cwd().parents)
if (d / "pipelines" / "eegpipe" / "__init__.py").exists()), None)
EEGPIPE = None
if _pipelines is not None:
sys.path.insert(0, str(_pipelines))
try:
import eegpipe as EEGPIPE
except Exception as exc:
print(f"pipelines/eegpipe found but not importable: {type(exc).__name__}: {exc}")
_out = tempfile.TemporaryDirectory(prefix="nb-c2-")
OUTPUT_DIR = Path(os.environ.get("EEG_COURSE_C2_OUT", _out.name))
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
available = l2.available_subjects("P3")
SUBSET = [f"sub-{i:03d}" for i in range(1, 11)]
SUBJECTS = available if FULL_COHORT else [s for s in SUBSET if s in available]
if EEGPIPE is not None:
CONFIG_PATH = _pipelines / "configs" / "erpcore-p3.yaml"
OVERRIDES = {"output.dir": str(OUTPUT_DIR), "subjects.include": SUBJECTS,
"steps.ica.iclabel.enabled": True}
config = EEGPIPE.load_config(CONFIG_PATH, overrides=OVERRIDES)
print(f"pipeline: pipelines/eegpipe {EEGPIPE.__version__}")
print(f" canonical order: {' -> '.join(EEGPIPE.CANONICAL_ORDER)}")
print(f" configuration: {CONFIG_PATH.relative_to(_pipelines.parent)} "
f"({len(CONFIG_PATH.read_text().splitlines())} lines)")
print(f" overrides: {{'output.dir': <temporary>, 'subjects.include': {len(SUBJECTS)} subjects, "
f"'steps.ica.iclabel.enabled': True}}")
print(f" name {config.name!r}, dataset {config.dataset.id} ({config.dataset.paradigm}), "
f"seed {config.seed}")
print(f" exclusions declared in the file: {config.subjects.exclude or 'none -- ds-erpcore documents no defective subjects'}")
else:
config = l2.PipelineConfig(subjects=tuple(SUBJECTS), seed=l2.SEED)
print("pipeline: helpers_l2.run_subject -- the interface site/CONTRACTS.md documents for "
"pipelines/eegpipe, used because that package is not importable here")
print()
print(f"cohort: {'FULL (all participants)' if FULL_COHORT else 'documented subset sub-001..sub-010'} -> "
f"{len(SUBJECTS)} subjects")
print(f" {', '.join(SUBJECTS)}")
print(f" the paradigm's OSF component lists {len(available)} subjects; already on this machine: "
f"{len(l2.cached_subjects('P3'))}")
print(f" output: a temporary directory (set EEG_COURSE_C2_OUT to keep the cleaned data and the reports)")
print(f" free disk: {helpers.free_disk_mb(OUTPUT_DIR) / 1000:.1f} GB")
In [3]:
if EEGPIPE is not None:
text = CONFIG_PATH.read_text(encoding="utf-8")
body = [l for l in text.splitlines() if not l.lstrip().startswith("#")]
print("\n".join(body)[:3200].rstrip())
print(" ... (the rest of the file is the reject step and the autoreject alternative)")
else:
print(config.to_yaml())
2. The entry point is raw BIDS¶
ERP CORE ships each paradigm as a per-subject BIDS-compatible folder: sub-XXX_task-P3_eeg.set/.fdt with _eeg.json, _channels.tsv, _events.tsv, _electrodes.tsv and _coordsystem.json beside it, plus dataset_description.json and participants.tsv at the top. What it does not have is the sub-XXX/eeg/ datatype directory that BIDS requires, so read_raw_bids cannot open it as it stands.
The cell below arranges the cached files into a valid BIDS tree using hard links, so not a byte of EEG is copied, and reads one subject back with read_raw_bids — the L2.1 skill applied to this capstone's own input. That read is the proof that the entry point is BIDS: the channel types and the line frequency come from the sidecars rather than from anybody's memory. The pipeline itself then reads the same files through its own loader.
In [4]:
from mne_bids import BIDSPath, read_raw_bids
BIDS_ROOT = OUTPUT_DIR / "bids"
src_root = l2.erpcore_cache() / "P3"
def link_or_copy(src: Path, dst: Path):
dst.parent.mkdir(parents=True, exist_ok=True)
if dst.exists():
return
try:
os.link(src, dst) # no bytes copied
except OSError:
shutil.copyfile(src, dst)
t0 = time.time()
for sid in SUBJECTS: # cached files are never re-fetched
l2.fetch_erpcore_subject("P3", sid)
print(f"{len(SUBJECTS)} subjects present in the cache ({time.time() - t0:.0f} s; "
f"{helpers.free_disk_mb(l2.erpcore_cache()) / 1000:.1f} GB free)")
for name in ("dataset_description.json", "participants.tsv", "participants.json", "README.txt",
"CHANGES", "task-P3_events.json"):
src = src_root / name
if src.exists():
link_or_copy(src, BIDS_ROOT / ("README" if name == "README.txt" else name))
linked = []
for sid in SUBJECTS:
d = src_root / sid
if d.is_dir():
for f in sorted(d.iterdir()):
if f.is_file():
link_or_copy(f, BIDS_ROOT / sid / "eeg" / f.name)
linked.append(sid)
print(f"BIDS tree built for {len(linked)} subjects with hard links -- no EEG data copied "
f"({sum(f.stat().st_size for f in BIDS_ROOT.rglob('*') if f.is_file()) / 1e6:.0f} MB linked, "
f"{helpers.free_disk_mb(OUTPUT_DIR) / 1000:.1f} GB still free)")
for line in sorted(str(p.relative_to(BIDS_ROOT)) for p in BIDS_ROOT.rglob("*") if p.is_file())[:10]:
print(f" {line}")
print(" ...")
BIDS_OK, bids_note = False, ""
try:
bp = BIDSPath(subject=SUBJECTS[0].replace("sub-", ""), task="P3", datatype="eeg", root=BIDS_ROOT,
suffix="eeg", extension=".set")
with warnings.catch_warnings():
warnings.simplefilter("ignore")
raw_bids = read_raw_bids(bp, verbose=False)
BIDS_OK = True
types = {t: raw_bids.get_channel_types().count(t) for t in sorted(set(raw_bids.get_channel_types()))}
bids_note = (f"read_raw_bids opened {SUBJECTS[0]}: {types} at {raw_bids.info['sfreq']:g} Hz, "
f"{len(raw_bids.annotations)} annotations, PowerLineFrequency "
f"{raw_bids.info.get('line_freq')} Hz -- every one of those read from a sidecar, not assumed")
except Exception as e:
bids_note = (f"read_raw_bids could not open the tree ({type(e).__name__}: {str(e)[:140]}); the pipeline "
"reads the EEGLAB files directly with the sidecars alongside")
print()
print(bids_note)
3. Run the cohort unattended¶
One call. A failure is a row in the report, never an exception that stops the loop — the rubric's first line. Each subject leaves four files behind: the cleaned continuous recording, the surviving epochs, the machine-readable run log, and the QC page.
In [5]:
def norm(res):
"""One shape for eegpipe.RunResult and for helpers_l2.run_subject's dict."""
if hasattr(res, "status"):
per = (res.rejection or {}).get("per_condition", {})
return {"subject": res.subject, "ok": res.status == "ok", "error": res.error,
"bads": {ch: v.get("criteria", []) for ch, v in (res.bad_channels or {}).items()},
"interpolated": list(res.interpolated or []), "rank": res.rank,
"rank_why": res.ica.get("rank_why", ""), "filter": res.filter or {},
"removed": [{"component": e["index"], "class": e["label"],
"probability": e.get("score", float("nan")), "evidence": e["reason"]}
for e in res.ica.get("excluded", [])],
"rank_after": res.ica.get("rank_after"),
"per_condition": {c: v["percent_rejected"] for c, v in per.items()},
"kept": {c: v["n_kept"] for c, v in per.items()},
"percent_rejected": (res.rejection or {}).get("percent_rejected", float("nan")),
"imbalance_pp": (res.rejection or {}).get("condition_imbalance_pp", float("nan")),
"criterion": (res.rejection or {}).get("criterion", {}),
"steps": [{"name": s["name"], "duration_s": s["duration_s"]} for s in res.steps],
"flags": list(res.flags or []), "outputs": dict(res.outputs or {}),
"config_hash": res.config_hash, "seed": res.seed, "versions": res.versions,
"duration_s": res.duration_s, "obj": res}
d = res
tab = {r["condition"]: r for r in d.get("rejection", {}).get("table", [])}
return {"subject": d["subject"], "ok": d["ok"], "error": d["error"],
"bads": d.get("bad_channels", {}).get("detection", {}).get("by_channel", {}),
"interpolated": d.get("bad_channels", {}).get("decision", {}).get("bads", []),
"rank": d.get("rank", {}).get("rank"), "rank_why": d.get("rank", {}).get("arithmetic", ""),
"filter": d.get("filter_resolved", {}), "removed": d.get("ica", {}).get("removed", []),
"rank_after": d.get("ica", {}).get("rank_after"),
"per_condition": {c: tab[c]["percent_rejected"] for c in ("target", "standard") if c in tab},
"kept": {c: tab[c]["n_kept"] for c in ("target", "standard") if c in tab},
"percent_rejected": tab.get("all", {}).get("percent_rejected", float("nan")),
"imbalance_pp": (tab["target"]["percent_rejected"] - tab["standard"]["percent_rejected"]
if "target" in tab and "standard" in tab else float("nan")),
"criterion": {"peak_to_peak_uv": d.get("rejection", {}).get("threshold_uv")},
"steps": d["log"].steps, "flags": [], "outputs": {}, "config_hash": d.get("config_hash"),
"seed": d["config"].get("seed"), "versions": d.get("versions", {}),
"duration_s": d.get("duration_s", float("nan")), "obj": d}
t_cohort = time.time()
if EEGPIPE is not None:
raw_results = EEGPIPE.run_cohort(config, SUBJECTS, progress=False)
else:
raw_results = [l2.run_subject(config, sid, verbose=False) for sid in SUBJECTS]
R = {}
for res in raw_results:
r = norm(res)
R[r["subject"]] = r
if not r["ok"]:
print(f" {r['subject']}: FAILED -- {str(r['error'])[:110]}")
continue
print(f" {r['subject']}: {len(r['interpolated'])} interpolated {r['interpolated'] or ''}, "
f"rank {r['rank']}, {len(r['removed'])} components removed, "
f"{r['percent_rejected']:.1f} % of epochs rejected "
f"(imbalance {r['imbalance_pp']:+.1f} pp), {r['duration_s']:.0f} s"
+ (f" flags: {r['flags']}" if r["flags"] else ""))
COHORT_S = time.time() - t_cohort
OK = [r for r in R.values() if r["ok"]]
files = [p for p in OUTPUT_DIR.rglob("*") if p.is_file() and "bids" not in p.parts]
print(f"\n{len(SUBJECTS)} subjects in {COHORT_S:.0f} s ({COHORT_S / max(len(SUBJECTS), 1):.0f} s each); "
f"{len(SUBJECTS) - len(OK)} failure(s)")
print(f"written: {len([f for f in files if f.name.endswith('_raw.fif')])} cleaned recordings, "
f"{len([f for f in files if f.name.endswith('_epo.fif')])} epoch files, "
f"{len([f for f in files if f.suffix == '.html'])} QC pages, "
f"{len([f for f in files if f.name == 'run-log.json'])} run logs -- "
f"{sum(f.stat().st_size for f in files) / 1e6:.0f} MB in total")
4. The group summary¶
In [6]:
# The subject-exclusion rule, fixed in the configuration file before any data were seen and applied here
# blind to any effect: a condition may not lose more than max_rejected_fraction of its trials, and a
# condition must keep at least MIN_TRIALS_PER_CONDITION.
MAX_REJECTED_PCT = (config.steps.reject.max_rejected_fraction * 100 if EEGPIPE is not None
else config.max_percent_rejected)
MIN_TRIALS_PER_CONDITION = 20
group = []
for sid in SUBJECTS:
r = R[sid]
worst = max([v for v in r["per_condition"].values()] or [float("nan")])
fewest = min([v for v in r["kept"].values()] or [0])
r["excluded"] = bool(not r["ok"] or worst > MAX_REJECTED_PCT or fewest < MIN_TRIALS_PER_CONDITION)
r["exclusion_reason"] = ("did not complete" if not r["ok"] else
f"{worst:.0f} % of a condition rejected (limit {MAX_REJECTED_PCT:.0f} %)"
if worst > MAX_REJECTED_PCT else
f"only {fewest} trials kept in a condition (minimum {MIN_TRIALS_PER_CONDITION})"
if fewest < MIN_TRIALS_PER_CONDITION else "")
group.append({"subject": sid, "ok": r["ok"],
"bads": r["interpolated"] or ["-"],
"rank after cleaning": r["rank"] if r["ok"] else float("nan"),
"ICA removed": len(r["removed"]) if r["ok"] else float("nan"),
"% rejected target": r["per_condition"].get("target", float("nan")),
"% rejected standard": r["per_condition"].get("standard", float("nan")),
"imbalance (pp)": r["imbalance_pp"],
"target kept": r["kept"].get("target", float("nan")),
"standard kept": r["kept"].get("standard", float("nan")),
"excluded": r["excluded"], "why": r["exclusion_reason"] or "-",
"flags": len(r["flags"]), "seconds": r["duration_s"]})
print(l2.fmt_table(group, list(group[0]), floatfmt="{:.2f}"))
print()
KEPT = [r for r in R.values() if r["ok"] and not r["excluded"]]
print(f"Subject-exclusion rule, fixed in the configuration before any data were seen and applied blind to any "
f"effect: a condition may lose at most {MAX_REJECTED_PCT:.0f} % of its trials and must keep at least "
f"{MIN_TRIALS_PER_CONDITION}. {len(KEPT)} of {len(SUBJECTS)} subjects pass.")
for r in R.values():
if r["excluded"]:
print(f" excluded: {r['subject']} -- {r['exclusion_reason']}")
print()
n_bads = [len(r["interpolated"]) for r in OK]
n_removed = [len(r["removed"]) for r in OK]
print(f"{len(OK)} of {len(SUBJECTS)} subjects completed.")
print(f"Channels interpolated: median {np.median(n_bads):.0f}, range {min(n_bads)}-{max(n_bads)} of 30.")
print(f"ICA components removed: median {np.median(n_removed):.0f}, range {min(n_removed)}-{max(n_removed)} "
f"of a rank of about {int(np.median([r['rank'] for r in OK]))} -- comparable across subjects is what "
"pf-overcleaning-ica asks for.")
print(f"Data rejected: target median {np.median([r['per_condition'].get('target', np.nan) for r in OK]):.1f} %, "
f"standard median {np.median([r['per_condition'].get('standard', np.nan) for r in OK]):.1f} %.")
print("Flags raised by the pipeline (recorded, never fatal):")
for r in OK:
if r["flags"]:
print(f" {r['subject']}: " + "; ".join(str(f) for f in r["flags"]))
n_conv = len([r for r in OK if any("converge" in str(f) for f in r["flags"])])
if n_conv:
print(f"\n{n_conv} of {len(OK)} subjects raise an ICA convergence flag at the configuration's iteration "
"limit. The decompositions are still usable -- the classifier labels them confidently and the "
"removed components are the expected ones -- but a run that reports this on most of its cohort is "
"telling you to raise the limit and re-run before publishing anything from it. That is what a flag "
"is for: it is data, not a crash, and it is in every subject's report.")
if EEGPIPE is not None:
print()
print("The package's own summary (capstone deliverable 4), written as summary.csv and summary.json:")
print(l2.fmt_table(EEGPIPE.summarize(raw_results), floatfmt="{:.2f}"))
5. The rubric's central check: was the rejection condition-biased?¶
The criterion is one number applied to every condition, fixed in the configuration file before any data were seen, so the condition played no part in setting it. That does not guarantee equal rejection — it guarantees only that the experimenter did not choose the imbalance. The check is to look.
In [7]:
gaps = np.array([r["imbalance_pp"] for r in OK], float)
print(f"per-condition imbalance (percentage points), {len(gaps)} subjects:")
print(f" median {np.nanmedian(gaps):+.1f} pp, mean {np.nanmean(gaps):+.1f} pp, "
f"range {np.nanmin(gaps):+.1f} to {np.nanmax(gaps):+.1f} pp")
for level in (20, 10):
who = [r["subject"] for r in OK if abs(r["imbalance_pp"]) >= level]
print(f" subjects with |imbalance| >= {level} pp: {who or 'none'}")
print(f" rejection criterion in force: {OK[0]['criterion']}")
print()
print("A difference of a few percentage points is life. A subject at 20 points is a finding about the "
"pipeline, not about the brain (L2.5), and it belongs in the report whether or not it changes the "
"conclusion.")
fig, axes = plt.subplots(1, 3, figsize=(15, 4.0))
labels = [r["subject"].replace("sub-", "") for r in OK]
axes[0].bar(range(len(OK)), [r["per_condition"].get("target", np.nan) for r in OK], width=0.4,
align="edge", color="tab:blue", label="target")
axes[0].bar([i + 0.4 for i in range(len(OK))], [r["per_condition"].get("standard", np.nan) for r in OK],
width=0.4, align="edge", color="tab:orange", label="standard")
axes[0].set(xticks=[i + 0.4 for i in range(len(OK))], ylabel="Trials rejected (%)",
title="Data rejected per condition, per subject (%)")
axes[0].set_xticklabels(labels, rotation=90, fontsize=7)
axes[0].legend(fontsize=8); axes[0].grid(alpha=0.3, axis="y")
axes[1].axhline(0, color="gray", lw=0.6)
for y in (20, -20):
axes[1].axhline(y, color="tab:red", lw=0.8, ls="--")
axes[1].bar(range(len(OK)), gaps, color=["tab:red" if abs(g) >= 20 else "tab:blue" for g in gaps])
axes[1].set(xticks=range(len(OK)), ylabel="target - standard (percentage points)",
title="Condition imbalance per subject (percentage points); dashed lines +/- 20 pp")
axes[1].set_xticklabels(labels, rotation=90, fontsize=7)
axes[1].grid(alpha=0.3, axis="y")
CH, WINDOW = l2.P3_CHANNEL, l2.P3_WINDOW
diffs, amps = [], []
for r in KEPT:
ep = getattr(r["obj"], "epochs", None) if hasattr(r["obj"], "status") else r["obj"].get("epochs")
if ep is None:
p = r["outputs"].get("epochs")
ep = mne.read_epochs(p, verbose=False) if p and Path(p).exists() else None
if ep is None or CH not in ep.ch_names:
continue
d = l2.difference_wave(ep)
diffs.append(d)
amps.append({"subject": r["subject"], f"{CH} mean (uV)": l2.mean_amplitude(d, CH, WINDOW),
"trials": int(d.nave)})
axes[2].plot(d.times * 1000, d.data[d.ch_names.index(CH)] * 1e6, lw=0.6, color="0.7")
for r in R.values(): # the excluded subjects, drawn but not averaged
if not r["excluded"] or not r["ok"]:
continue
ep = getattr(r["obj"], "epochs", None) if hasattr(r["obj"], "status") else r["obj"].get("epochs")
if ep is None:
p = r["outputs"].get("epochs")
ep = mne.read_epochs(p, verbose=False) if p and Path(p).exists() else None
if ep is not None and CH in ep.ch_names:
d = l2.difference_wave(ep)
axes[2].plot(d.times * 1000, d.data[d.ch_names.index(CH)] * 1e6, lw=0.7, color="tab:red", ls=":",
label=f"{r['subject']} (excluded)")
if diffs:
grand = mne.grand_average(diffs)
axes[2].plot(grand.times * 1000, grand.data[grand.ch_names.index(CH)] * 1e6, lw=2.0, color="tab:blue",
label=f"grand average, n = {len(diffs)}")
axes[2].legend(fontsize=8)
axes[2].axvspan(WINDOW[0] * 1000, WINDOW[1] * 1000, color="tab:orange", alpha=0.18)
axes[2].axhline(0, color="gray", lw=0.6); axes[2].axvline(0, color="gray", lw=0.6)
axes[2].set(xlabel="Time from stimulus (ms)", ylabel="Amplitude (uV)",
title=f"Target minus standard at {CH} (uV, positive up; dotted red = excluded by the rule)")
axes[2].grid(alpha=0.3)
fig.tight_layout()
plt.show() # render the static figure(s) of this cell inline
if amps:
print(l2.fmt_table(amps, list(amps[0]), floatfmt="{:+.2f}"))
v = np.array([a[f"{CH} mean (uV)"] for a in amps])
print(f"\ngrand mean over the {len(v)} retained subjects: {v.mean():+.2f} uV "
f"(SD {v.std(ddof=1):.2f}, SEM {v.std(ddof=1) / np.sqrt(len(v)):.2f}, "
f"positive in {int((v > 0).sum())}/{len(v)} subjects). The excluded subjects are drawn in the "
"figure and left out of the average, which is the whole point of fixing the rule in advance.")
6. One QC report, as a stranger would read it¶
In [8]:
SHOW = OK[0]["subject"]
qc_path = R[SHOW]["outputs"].get("qc")
if qc_path and Path(qc_path).exists():
html = Path(qc_path).read_text(encoding="utf-8")
print(f"{SHOW}: {Path(qc_path).name}, {len(html) / 1000:.0f} kB of self-contained HTML; the same page "
f"exists for every subject, and index.html links them all")
else:
html = l2.qc_report_html(R[SHOW]["obj"], title=f"QC report -- {SHOW} (ds-erpcore P3)")
print(f"{SHOW}: rendered by helpers_l2.qc_report_html, {len(html) / 1000:.0f} kB")
display(HTML(html))
QC report — sub-001 ok
ds-erpcore · configuration erpcore-p3
(hash 06fd3f5c3717) · seed 20260917
· 2026-09-18T14:25:59+00:00 · 10.3 s
Read this first — what would make you distrust this subject
- ICA did not converge (500 iterations)
0bad channels
0interpolated
24rank
5ICA removed
191epochs kept
4.5% rejected
Bad channels
30 EEG channels, 93 windows of 5.0 s. Criteria and thresholds are in the configuration below; detection ran before filtering and before re-referencing.
(nothing to show)
Rank after interpolation and re-referencing: 24 — 0 interpolated channel(s) (none) and the reference each cost one. This is the rank the ICA step was given.
Filter settings, as resolved
| what | high-pass (Hz) | low-pass (Hz) | method | phase | length (samples) | low transition | high transition |
|---|---|---|---|---|---|---|---|
| analysis data | 0.10 | 30.00 | fir | zero | 33793 | auto | auto |
| ICA copy (fit only) | 1.00 | 30.00 | fir | zero | 3381 | — | — |
- Resampled to 256.0 Hz — after the low-pass so nothing above the new Nyquist can alias, and before epoching so events become sample indices only once
- Recorded passband after filtering: 0.10–30.0 Hz
- highpass: 0.1 Hz: high enough to remove the drift of a 405-s recording, low enough not to distort the P3's late positivity (L2.4, pf-hp-cutoff-erp).
- lowpass: 30 Hz: the P3 is a slow component, and 30 Hz keeps the 60 Hz mains and the muscle band out without touching it. No notch is needed behind a 30 Hz low-pass.
- resample: 1024 Hz costs time and disk for no gain at these cutoffs; 256 Hz is still eight times the low-pass. Resampling happens before epoching so events are converted once.
- ica_highpass: 1 Hz for the ICA fit only: drift dominates the variance and costs components; the unmixing is applied to the 0.1 Hz analysis data (L2.4, L2.6).
- ICA copy: ICA is fitted on a 1 Hz high-passed copy because slow drift dominates the variance and costs components; the unmixing is applied to the analysis data (L2.4)
- filtered on continuous data, so edge artifacts sit at the ends of the recording rather than inside every epoch (L2.4)
ICA components removed
infomax, 29 components, fitted on 1 Hz high-passed copy, seed 20260917, 500 iterations, converged: no. Components requested: rank carried through the pipeline: 30 EEG channels - 0 interpolated - reference cost = 29.
| component | class | score | ICLabel | evidence |
|---|---|---|---|---|
| IC8 | eye blink | 0.97 | eye blink p=0.971 | ICLabel classified it eye blink with probability 0.97 (threshold 0.8) |
| IC3 | muscle artifact | 0.96 | muscle artifact p=0.964 | ICLabel classified it muscle artifact with probability 0.96 (threshold 0.8) |
| IC27 | muscle artifact | 0.94 | muscle artifact p=0.944 | ICLabel classified it muscle artifact with probability 0.94 (threshold 0.8) |
| IC10 | eye blink | 0.82 | eye blink p=0.82 | ICLabel classified it eye blink with probability 0.82 (threshold 0.8) |
| IC0 | eye | 0.66 | eye blink p=0.986 | |correlation| with HEOG_left, HEOG_right, VEOG_lower = 0.66, above the configured threshold 0.5 (measure: correlation) |
5 component(s) removed; rank after cleaning 24.
Data rejected, per condition
| condition | epochs before | kept | rejected | % rejected |
|---|---|---|---|---|
| target | 40 | 38 | 2 | 5.00 |
| standard | 160 | 153 | 7 | 4.38 |
- Method:
threshold - Criterion (the same for every condition): peak-to-peak 100.0 µV, flat 1.0 µV
- Overall: 4.5% of epochs rejected; per-condition spread 0.6 percentage points
- Reasons recorded by MNE: PO8: 7, P8: 3, CPz: 2, P7: 2, PO7: 2, Fp1: 1, C3: 1, P4: 1
Figures
Run log
| step | status | seconds | what it recorded |
|---|---|---|---|
| load | ok | 0.15 | {"dataset": "ds-erpcore", "duration_s": 467.0, "n_channels": 33, "sfreq_hz": 1024.0} |
| montage | ok | 0.01 | {"n_positioned": 30} |
| bad_channels | ok | 0.15 | {"bads": []} |
| filter | ok | 1.23 | {"resampled_to_hz": 256.0, "sfreq_out_hz": 256.0} |
| interpolate | skipped | 0.00 | {"reason": "no bad channels to interpolate"} |
| reference | ok | 0.01 | {"n_channels": 33, "rank_after": 29, "type": "average"} |
| ica | ok | 8.72 | {"n_excluded": 5, "rank_after": 24} |
| epoch | ok | 0.04 | {"n_epochs": 200, "n_epochs_per_condition": {"standard": 160, "target": 40}, "sfreq_hz": 256.0} |
| reject | ok | 0.03 | {"percent_rejected": 4.5} |
The full run log (JSON) — every parameter as resolved, every note
{
"subject": "sub-001",
"dataset": "ds-erpcore",
"config_name": "erpcore-p3",
"config_hash": "06fd3f5c3717",
"seed": 20260917,
"status": "ok",
"error": null,
"traceback": null,
"started_at": "2026-09-18T14:25:59+00:00",
"duration_s": 10.327,
"versions": {
"python": "3.13.5",
"platform": "Darwin 25.3.0 (arm64)",
"mne": "1.10.2",
"numpy": "2.1.3",
"scipy": "1.15.3",
"matplotlib": "3.10.6",
"scikit-learn": "1.6.1",
"autoreject": "0.5.0",
"mne-icalabel": "0.9.0",
"python-picard": null,
"PyYAML": "6.0.2",
"pinned": {
"mne": "1.10.2",
"numpy": ">=2.1,<3",
"scipy": ">=1.15,<2"
},
"on_mismatch": "warn"
},
"version_mismatches": [],
"bad_channels": {},
"interpolated": [],
"rank": 24,
"filter": {
"sfreq_in_hz": 1024.0,
"ica_copy": {
"l_freq_hz": 1.0,
"h_freq_hz": 30.0,
"filter_length_samples": 3381,
"filter_length_s": 3.3018,
"why": "ICA is fitted on a 1 Hz high-passed copy because slow drift dominates the variance and costs components; the unmixing is applied to the analysis data (L2.4)"
},
"analysis": {
"l_freq_hz": 0.1,
"h_freq_hz": 30.0,
"method": "fir",
"phase": "zero",
"fir_design": "firwin",
"fir_window": "hamming",
"l_trans_bandwidth": "auto",
"h_trans_bandwidth": "auto",
"filter_length_samples": 33793,
"filter_length_s": 33.001
},
"resampled_to_hz": 256.0,
"resample_note": "after the low-pass so nothing above the new Nyquist can alias, and before epoching so events become sample indices only once",
"info_highpass_hz": 0.1,
"info_lowpass_hz": 30.0,
"sfreq_out_hz": 256.0,
"justification": {
"highpass": "0.1 Hz: high enough to remove the drift of a 405-s recording, low enough not to distort the P3's late positivity (L2.4, pf-hp-cutoff-erp).\n",
"lowpass": "30 Hz: the P3 is a slow component, and 30 Hz keeps the 60 Hz mains and the muscle band out without touching it. No notch is needed behind a 30 Hz low-pass.\n",
"resample": "1024 Hz costs time and disk for no gain at these cutoffs; 256 Hz is still eight times the low-pass. Resampling happens before epoching so events are converted once.\n",
"ica_highpass": "1 Hz for the ICA fit only: drift dominates the variance and costs components; the unmixing is applied to the 0.1 Hz analysis data (L2.4, L2.6).\n"
},
"edge_note": "filtered on continuous data, so edge artifacts sit at the ends of the recording rather than inside every epoch (L2.4)"
},
"ica": {
"method": "infomax",
"fit_params": {
"extended": true
},
"seed": 20260917,
"n_components": 29,
"n_components_requested": 29,
"rank_why": "rank carried through the pipeline: 30 EEG channels - 0 interpolated - reference cost = 29",
"fitted_on": "1 Hz high-passed copy",
"n_iterations": 500,
"converged": false,
"excluded": [
{
"index": 8,
"label": "eye blink",
"score": 0.971,
"reason": "ICLabel classified it eye blink with probability 0.97 (threshold 0.8)",
"detector": "mne-icalabel",
"iclabel": {
"label": "eye blink",
"probability": 0.971
}
},
{
"index": 3,
"label": "muscle artifact",
"score": 0.964,
"reason": "ICLabel classified it muscle artifact with probability 0.96 (threshold 0.8)",
"detector": "mne-icalabel",
"iclabel": {
"label": "muscle artifact",
"probability": 0.964
}
},
{
"index": 27,
"label": "muscle artifact",
"score": 0.944,
"reason": "ICLabel classified it muscle artifact with probability 0.94 (threshold 0.8)",
"detector": "mne-icalabel",
"iclabel": {
"label": "muscle artifact",
"probability": 0.944
}
},
{
"index": 10,
"label": "eye blink",
"score": 0.82,
"reason": "ICLabel classified it eye blink with probability 0.82 (threshold 0.8)",
"detector": "mne-icalabel",
"iclabel": {
"label": "eye blink",
"probability": 0.82
}
},
{
"index": 0,
"label": "eye",
"score": 0.664,
"reason": "|correlation| with HEOG_left, HEOG_right, VEOG_lower = 0.66, above the configured threshold 0.5 (measure: correlation)",
"detector": "ICA.find_bads_eog(EOG channels in the recording)",
"iclabel": {
"label": "eye blink",
"probability": 0.986
}
}
],
"n_excluded": 5,
"kept_by_max_remove": [],
"applied": true,
"rank_before": 29,
"rank_after": 24,
"eog": {
"enabled": true,
"channels": [
"HEOG_left",
"HEOG_right",
"VEOG_lower"
],
"provenance": "EOG channels in the recording"
},
"iclabel": {
"enabled": true,
"labels": [
"eye blink",
"other",
"brain",
"muscle artifact",
"eye blink",
"brain",
"other",
"brain",
"eye blink",
"brain",
"eye blink",
"muscle artifact",
"eye blink",
"brain",
"brain",
"brain",
"brain",
"brain",
"brain",
"other",
"brain",
"other",
"other",
"brain",
"brain",
"other",
"other",
"muscle artifact",
"brain"
],
"threshold": 0.8,
"requirements": "extended infomax, 1-100 Hz, average reference (mne-icalabel's stated conditions)"
}
},
"epoching": {
"n_epochs_per_condition": {
"target": 40,
"standard": 160
},
"dropped_at_epoching": {},
"conditions": {
"target": {
"requested": [
11,
22,
33,
44,
55
],
"codes": [
1,
9,
15,
21,
27
],
"not_found": [],
"n_events": 40
},
"standard": {
"requested": [
12,
13,
14,
15,
21,
23,
24,
25,
31,
32,
34,
35,
41,
42,
43,
45,
51,
52,
53,
54
],
"codes": [
2,
3,
4,
5,
8,
10,
11,
12,
13,
14,
16,
17,
18,
19,
20,
22,
23,
24,
25,
26
],
"not_found": [],
"n_events": 160
}
}
},
"rejection": {
"method": "threshold",
"n_epochs_before": 200,
"criterion": {
"peak_to_peak_uv": 100.0,
"flat_uv": 1.0,
"eog_peak_to_peak_uv": null
},
"criterion_note": "one criterion for every condition, set before the data were looked at (pf-condition-biased-rejection)",
"per_condition": {
"target": {
"n_before": 40,
"n_kept": 38,
"n_rejected": 2,
"percent_rejected": 5.0
},
"standard": {
"n_before": 160,
"n_kept": 153,
"n_rejected": 7,
"percent_rejected": 4.38
}
},
"n_epochs_after": 191,
"percent_rejected": 4.5,
"condition_imbalance_pp": 0.62,
"drop_reasons": {
"CPz": 2,
"Fp1": 1,
"P7": 2,
"PO7": 2,
"P8": 3,
"PO8": 7,
"C3": 1,
"P4": 1,
"Fp2": 1,
"PO3": 1,
"Oz": 1
},
"flags": []
},
"facts": {
"dataset": "ds-erpcore",
"subject": "sub-001",
"paradigm": "P3",
"name": "ERP CORE (Compendium of Open Resources and Experiments)",
"license": "CC BY 4.0 (OSF node thsqg; TODO(confirm): the component's own LICENSE file says CC BY-SA 4.0 and its dataset_description.json says CC0 -- the author reconciles this, spec section 10.11 item 8)",
"source": "https://osf.io/thsqg/",
"source_doi": "TODO(confirm)",
"citation": "Kappenman, Farrens, Zhang, Stewart & Luck (2021), ERP CORE",
"sfreq_hz": 1024.0,
"n_channels": 33,
"n_channels_note": "30 EEG + 3 EOG, 10-20 placement (sub-002 P3 channels.tsv)",
"reference": "CMS (Biosemi ActiveTwo online reference)",
"hardware_filters": "none recorded (BIDS eeg.json SoftwareFilters: n/a)",
"mains_hz": 60,
"subjects": 40,
"access": "open",
"files": [
"sub-001_task-P3_events.tsv",
"sub-001_task-P3_eeg.json",
"sub-001_task-P3_channels.tsv",
"sub-001_task-P3_electrodes.tsv",
"sub-001_task-P3_coordsystem.json",
"sub-001_task-P3_eeg.set",
"sub-001_task-P3_eeg.fdt"
],
"source_kind": "data/scripts/fetch_erpcore.py",
"sidecar": {
"subject": "sub-001",
"eeg_json": {
"TaskName": "P3",
"Manufacturer": "Biosemi",
"ManufacturersModelName": "ActiveTwo",
"EEGReference": "CMS",
"SamplingFrequency": 1024,
"PowerLineFrequency": 60,
"SoftwareFilters": "n/a",
"EEGPlacementScheme": "10-20",
"RecordingType": "continuous",
"EEGChannelCount": 30,
"EOGChannelCount": 3,
"RecordingDuration": 467,
"TaskDescription": "The P3 was elicited in an active visual oddball task adapted from Luck et al. (2009). The letters A, B, C, D, and E were presented in random order (p = .2 for each letter). One letter was designated the target for a given block of trials, and the other 4 letters were non-targets. Thus, the probability of the target category was .2, but the same physical stimulus served as a target in some blocks and a nontarget in others. Participants responded whether the letter presented on each trial was the target or a non-target for that block."
},
"channels": [
{
"name": "FP1",
"type": "EEG",
"units": "microV"
},
{
"name": "F3",
"type": "EEG",
"units": "microV"
},
{
"name": "F7",
"type": "EEG",
"units": "microV"
},
{
"name": "FC3",
"type": "EEG",
"units": "microV"
},
{
"name": "C3",
"type": "EEG",
"units": "microV"
},
{
"name": "C5",
"type": "EEG",
"units": "microV"
},
{
"name": "P3",
"type": "EEG",
"units": "microV"
},
{
"name": "P7",
"type": "EEG",
"units": "microV"
},
{
"name": "P9",
"type": "EEG",
"units": "microV"
},
{
"name": "PO7",
"type": "EEG",
"units": "microV"
},
{
"name": "PO3",
"type": "EEG",
"units": "microV"
},
{
"name": "O1",
"type": "EEG",
"units": "microV"
},
{
"name": "Oz",
"type": "EEG",
"units": "microV"
},
{
"name": "Pz",
"type": "EEG",
"units": "microV"
},
{
"name": "CPz",
"type": "EEG",
"units": "microV"
},
{
"name": "FP2",
"type": "EEG",
"units": "microV"
},
{
"name": "Fz",
"type": "EEG",
"units": "microV"
},
{
"name": "F4",
"type": "EEG",
"units": "microV"
},
{
"name": "F8",
"type": "EEG",
"units": "microV"
},
{
"name": "FC4",
"type": "EEG",
"units": "microV"
},
{
"name": "FCz",
"type": "EEG",
"units": "microV"
},
{
"name": "Cz",
"type": "EEG",
"units": "microV"
},
{
"name": "C4",
"type": "EEG",
"units": "microV"
},
{
"name": "C6",
"type": "EEG",
"units": "microV"
},
{
"name": "P4",
"type": "EEG",
"units": "microV"
},
{
"name": "P8",
"type": "EEG",
"units": "microV"
},
{
"name": "P10",
"type": "EEG",
"units": "microV"
},
{
"name": "PO8",
"type": "EEG",
"units": "microV"
},
{
"name": "PO4",
"type": "EEG",
"units": "microV"
},
{
"name": "O2",
"type": "EEG",
"units": "microV"
},
{
"name": "HEOG_left",
"type": "EOG",
"units": "microV"
},
{
"name": "HEOG_right",
"type": "EOG",
"units": "microV"
},
{
"name": "VEOG_lower",
"type": "EOG",
"units": "microV"
}
],
"eeg_channels": [
"FP1",
"F3",
"F7",
"FC3",
"C3",
"C5",
"P3",
"P7",
"P9",
"PO7",
"PO3",
"O1",
"Oz",
"Pz",
"CPz",
"FP2",
"Fz",
"F4",
"F8",
"FC4",
"FCz",
"Cz",
"C4",
"C6",
"P4",
"P8",
"P10",
"PO8",
"PO4",
"O2"
],
"eog_channels": [
"HEOG_left",
"HEOG_right",
"VEOG_lower"
]
},
"montage": "standard_1005"
},
"flags": [
"ICA did not converge (500 iterations)"
],
"outputs": {
"config": ".../nb-c2-dfglqahu/resolved-config.yaml",
"raw": ".../sub-001/sub-001_desc-clean_raw.fif",
"epochs": ".../sub-001/sub-001_desc-clean_epo.fif"
},
"steps": [
{
"name": "load",
"params": {
"preload": true,
"eog_channels": [
"VEOG",
"HEOG"
],
"misc_channels": [],
"rename": {
"FP1": "Fp1",
"FP2": "Fp2"
},
"crop_s": null,
"bad_segments_s": [],
"dataset": {
"id": "ds-erpcore",
"paradigm": "P3",
"runs": [],
"path": null,
"cache_dir": null,
"synthetic": {}
},
"seed": 20260917
},
"duration_s": 0.1505,
"notes": {
"dataset": "ds-erpcore",
"subject": "sub-001",
"source": "ERP CORE (Compendium of Open Resources and Experiments)",
"license": "CC BY 4.0 (OSF node thsqg; TODO(confirm): the component's own LICENSE file says CC BY-SA 4.0 and its dataset_description.json says CC0 -- the author reconciles this, spec section 10.11 item 8)",
"files": [
"sub-001_task-P3_events.tsv",
"sub-001_task-P3_eeg.json",
"sub-001_task-P3_channels.tsv",
"sub-001_task-P3_electrodes.tsv",
"sub-001_task-P3_coordsystem.json",
"sub-001_task-P3_eeg.set",
"sub-001_task-P3_eeg.fdt"
],
"sfreq_hz": 1024.0,
"duration_s": 467.0,
"n_channels": 33,
"channel_counts": {
"eeg": 30,
"eog": 3,
"stim": 0
},
"channels_renamed": {
"FP1": "Fp1",
"FP2": "Fp2"
},
"channel_types_set": {
"HEOG_left": "eog",
"HEOG_right": "eog",
"VEOG_lower": "eog"
},
"header_highpass_hz": 0.0,
"header_lowpass_hz": 512.0,
"bads_in_file": [],
"n_annotations": 402,
"annotation_descriptions": [
"11",
"12",
"13",
"14",
"15",
"201",
"202",
"21",
"22",
"23",
"24",
"25",
"31",
"32",
"33",
"34",
"35",
"41",
"42",
"43"
],
"bad_segments_annotated": [],
"cropped_to_s": null,
"documented_defect": null,
"rank_start": 30
},
"status": "ok",
"error": null
},
{
"name": "montage",
"params": {
"name": "standard_1005",
"match_case": true,
"on_missing": "warn"
},
"duration_s": 0.0057,
"notes": {
"montage": "standard_1005",
"n_eeg": 30,
"n_positioned": 30,
"channels_without_position": [],
"note": "every EEG channel has a position"
},
"status": "ok",
"error": null
},
{
"name": "bad_channels",
"params": {
"enabled": true,
"window_s": 5.0,
"flat_uv": 1.0,
"flat_fraction": 0.5,
"deviation_z": 5.0,
"correlation_threshold": 0.4,
"correlation_fraction": 0.3,
"hf_noise_z": 5.0,
"manual": [],
"keep_existing": true,
"max_bad_fraction": 0.2
},
"duration_s": 0.1454,
"notes": {
"n_eeg": 30,
"n_windows": 93,
"window_s": 5.0,
"bads": [],
"bad_fraction": 0.0,
"criteria_fired": {},
"per_channel": {
"Fp1": {
"criteria": [],
"metrics": {
"std_uv": 1598.951,
"deviation_z": 1.52,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.85,
"hf_ratio": 0.003,
"hf_noise_z": -0.91
}
},
"F3": {
"criteria": [],
"metrics": {
"std_uv": 807.166,
"deviation_z": 0.29,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.973,
"hf_ratio": 0.004,
"hf_noise_z": -0.67
}
},
"F7": {
"criteria": [],
"metrics": {
"std_uv": 462.048,
"deviation_z": -0.72,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.863,
"hf_ratio": 0.015,
"hf_noise_z": 1.18
}
},
"FC3": {
"criteria": [],
"metrics": {
"std_uv": 739.151,
"deviation_z": 0.13,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.976,
"hf_ratio": 0.004,
"hf_noise_z": -0.63
}
},
"C3": {
"criteria": [],
"metrics": {
"std_uv": 1260.008,
"deviation_z": 1.09,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.945,
"hf_ratio": 0.002,
"hf_noise_z": -1.02
}
},
"C5": {
"criteria": [],
"metrics": {
"std_uv": 662.041,
"deviation_z": -0.07,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.011,
"median_best_correlation": 0.874,
"hf_ratio": 0.009,
"hf_noise_z": 0.15
}
},
"P3": {
"criteria": [],
"metrics": {
"std_uv": 554.609,
"deviation_z": -0.39,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.011,
"median_best_correlation": 0.781,
"hf_ratio": 0.003,
"hf_noise_z": -0.79
}
},
"P7": {
"criteria": [],
"metrics": {
"std_uv": 741.678,
"deviation_z": 0.14,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.772,
"hf_ratio": 0.007,
"hf_noise_z": -0.12
}
},
"P9": {
"criteria": [],
"metrics": {
"std_uv": 157.485,
"deviation_z": -2.65,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.764,
"hf_ratio": 0.029,
"hf_noise_z": 3.65
}
},
"PO7": {
"criteria": [],
"metrics": {
"std_uv": 306.236,
"deviation_z": -1.46,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.022,
"median_best_correlation": 0.705,
"hf_ratio": 0.021,
"hf_noise_z": 2.35
}
},
"PO3": {
"criteria": [],
"metrics": {
"std_uv": 679.767,
"deviation_z": -0.02,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.032,
"median_best_correlation": 0.751,
"hf_ratio": 0.004,
"hf_noise_z": -0.66
}
},
"O1": {
"criteria": [],
"metrics": {
"std_uv": 695.603,
"deviation_z": 0.02,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.763,
"hf_ratio": 0.009,
"hf_noise_z": 0.19
}
},
"Oz": {
"criteria": [],
"metrics": {
"std_uv": 483.576,
"deviation_z": -0.63,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.789,
"hf_ratio": 0.009,
"hf_noise_z": 0.17
}
},
"Pz": {
"criteria": [],
"metrics": {
"std_uv": 108.066,
"deviation_z": -3.33,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.845,
"hf_ratio": 0.012,
"hf_noise_z": 0.78
}
},
"CPz": {
"criteria": [],
"metrics": {
"std_uv": 183.924,
"deviation_z": -2.37,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.924,
"hf_ratio": 0.009,
"hf_noise_z": 0.17
}
},
"Fp2": {
"criteria": [],
"metrics": {
"std_uv": 1926.533,
"deviation_z": 1.85,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.842,
"hf_ratio": 0.003,
"hf_noise_z": -0.82
}
},
"Fz": {
"criteria": [],
"metrics": {
"std_uv": 1274.242,
"deviation_z": 1.11,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.962,
"hf_ratio": 0.002,
"hf_noise_z": -1.03
}
},
"F4": {
"criteria": [],
"metrics": {
"std_uv": 429.606,
"deviation_z": -0.85,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.916,
"hf_ratio": 0.014,
"hf_noise_z": 1.12
}
},
"F8": {
"criteria": [],
"metrics": {
"std_uv": 945.054,
"deviation_z": 0.57,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.086,
"median_best_correlation": 0.662,
"hf_ratio": 0.015,
"hf_noise_z": 1.24
}
},
"FC4": {
"criteria": [],
"metrics": {
"std_uv": 847.671,
"deviation_z": 0.38,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.011,
"median_best_correlation": 0.883,
"hf_ratio": 0.006,
"hf_noise_z": -0.28
}
},
"FCz": {
"criteria": [],
"metrics": {
"std_uv": 490.412,
"deviation_z": -0.61,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.968,
"hf_ratio": 0.004,
"hf_noise_z": -0.62
}
},
"Cz": {
"criteria": [],
"metrics": {
"std_uv": 106.081,
"deviation_z": -3.36,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.947,
"hf_ratio": 0.017,
"hf_noise_z": 1.67
}
},
"C4": {
"criteria": [],
"metrics": {
"std_uv": 891.055,
"deviation_z": 0.47,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.906,
"hf_ratio": 0.004,
"hf_noise_z": -0.64
}
},
"C6": {
"criteria": [],
"metrics": {
"std_uv": 1534.857,
"deviation_z": 1.44,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.022,
"median_best_correlation": 0.862,
"hf_ratio": 0.005,
"hf_noise_z": -0.51
}
},
"P4": {
"criteria": [],
"metrics": {
"std_uv": 715.976,
"deviation_z": 0.07,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.854,
"hf_ratio": 0.005,
"hf_noise_z": -0.42
}
},
"P8": {
"criteria": [],
"metrics": {
"std_uv": 453.434,
"deviation_z": -0.75,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.83,
"hf_ratio": 0.01,
"hf_noise_z": 0.42
}
},
"P10": {
"criteria": [],
"metrics": {
"std_uv": 1462.528,
"deviation_z": 1.36,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.876,
"hf_ratio": 0.002,
"hf_noise_z": -0.93
}
},
"PO8": {
"criteria": [],
"metrics": {
"std_uv": 920.719,
"deviation_z": 0.53,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.829,
"hf_ratio": 0.009,
"hf_noise_z": 0.12
}
},
"PO4": {
"criteria": [],
"metrics": {
"std_uv": 362.411,
"deviation_z": -1.15,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.0,
"median_best_correlation": 0.75,
"hf_ratio": 0.014,
"hf_noise_z": 1.13
}
},
"O2": {
"criteria": [],
"metrics": {
"std_uv": 489.419,
"deviation_z": -0.61,
"flat_fraction": 0.0,
"low_correlation_fraction": 0.011,
"median_best_correlation": 0.822,
"hf_ratio": 0.012,
"hf_noise_z": 0.68
}
}
},
"exceeds_max_bad_fraction": false,
"flags": [],
"note": "detection ran before filtering and before re-referencing (L2.8): a bad channel that joins an average reference contaminates every channel"
},
"status": "ok",
"error": null
},
{
"name": "filter",
"params": {
"l_freq": 0.1,
"h_freq": 30.0,
"method": "fir",
"phase": "zero",
"fir_design": "firwin",
"fir_window": "hamming",
"l_trans_bandwidth": "auto",
"h_trans_bandwidth": "auto",
"iir_params": {},
"notch_freqs": [],
"notch_widths": null,
"resample_hz": 256.0,
"ica_l_freq": 1.0,
"ica_h_freq": null,
"justification": {
"highpass": "0.1 Hz: high enough to remove the drift of a 405-s recording, low enough not to distort the P3's late positivity (L2.4, pf-hp-cutoff-erp).\n",
"lowpass": "30 Hz: the P3 is a slow component, and 30 Hz keeps the 60 Hz mains and the muscle band out without touching it. No notch is needed behind a 30 Hz low-pass.\n",
"resample": "1024 Hz costs time and disk for no gain at these cutoffs; 256 Hz is still eight times the low-pass. Resampling happens before epoching so events are converted once.\n",
"ica_highpass": "1 Hz for the ICA fit only: drift dominates the variance and costs components; the unmixing is applied to the 0.1 Hz analysis data (L2.4, L2.6).\n"
}
},
"duration_s": 1.2255,
"notes": {
"sfreq_in_hz": 1024.0,
"ica_copy": {
"l_freq_hz": 1.0,
"h_freq_hz": 30.0,
"filter_length_samples": 3381,
"filter_length_s": 3.3018,
"why": "ICA is fitted on a 1 Hz high-passed copy because slow drift dominates the variance and costs components; the unmixing is applied to the analysis data (L2.4)"
},
"analysis": {
"l_freq_hz": 0.1,
"h_freq_hz": 30.0,
"method": "fir",
"phase": "zero",
"fir_design": "firwin",
"fir_window": "hamming",
"l_trans_bandwidth": "auto",
"h_trans_bandwidth": "auto",
"filter_length_samples": 33793,
"filter_length_s": 33.001
},
"resampled_to_hz": 256.0,
"resample_note": "after the low-pass so nothing above the new Nyquist can alias, and before epoching so events become sample indices only once",
"info_highpass_hz": 0.1,
"info_lowpass_hz": 30.0,
"sfreq_out_hz": 256.0,
"justification": {
"highpass": "0.1 Hz: high enough to remove the drift of a 405-s recording, low enough not to distort the P3's late positivity (L2.4, pf-hp-cutoff-erp).\n",
"lowpass": "30 Hz: the P3 is a slow component, and 30 Hz keeps the 60 Hz mains and the muscle band out without touching it. No notch is needed behind a 30 Hz low-pass.\n",
"resample": "1024 Hz costs time and disk for no gain at these cutoffs; 256 Hz is still eight times the low-pass. Resampling happens before epoching so events are converted once.\n",
"ica_highpass": "1 Hz for the ICA fit only: drift dominates the variance and costs components; the unmixing is applied to the 0.1 Hz analysis data (L2.4, L2.6).\n"
},
"edge_note": "filtered on continuous data, so edge artifacts sit at the ends of the recording rather than inside every epoch (L2.4)"
},
"status": "ok",
"error": null
},
{
"name": "interpolate",
"params": {
"enabled": true,
"method": "spline",
"reset_bads": true,
"exclude": [],
"max_interpolate": 6
},
"duration_s": 0.0,
"notes": {
"reason": "no bad channels to interpolate",
"bads_left_marked": [],
"rank": 30
},
"status": "skipped",
"error": null
},
{
"name": "reference",
"params": {
"type": "average",
"channels": [],
"projection": false,
"add_channels": []
},
"duration_s": 0.0135,
"notes": {
"type": "average",
"projection": false,
"note": "the average reference is a subtraction of the mean over channels, so the data lose one rank; the ICA step is told",
"rank_before": 30,
"rank_after": 29,
"rank_cost": 1,
"n_channels": 33
},
"status": "ok",
"error": null
},
{
"name": "ica",
"params": {
"enabled": true,
"method": "infomax",
"fit_params": {
"extended": true
},
"n_components": "rank",
"max_iter": 500,
"decim": 3,
"seed": 20260917,
"reject_by_annotation": true,
"fit_on": "ica_copy",
"eog": {
"enabled": true,
"channels": [],
"measure": "correlation",
"threshold": 0.5
},
"muscle": {
"enabled": false,
"threshold": 0.8
},
"iclabel": {
"enabled": true,
"labels": [
"eye blink",
"muscle artifact",
"heart beat",
"line noise",
"channel noise"
],
"threshold": 0.8
},
"max_remove": 6,
"apply": true
},
"duration_s": 8.7166,
"notes": {
"method": "infomax",
"fit_params": {
"extended": true
},
"seed": 20260917,
"n_components": 29,
"n_components_requested": 29,
"rank_why": "rank carried through the pipeline: 30 EEG channels - 0 interpolated - reference cost = 29",
"fitted_on": "1 Hz high-passed copy",
"n_iterations": 500,
"converged": false,
"excluded": [
{
"index": 8,
"label": "eye blink",
"score": 0.971,
"reason": "ICLabel classified it eye blink with probability 0.97 (threshold 0.8)",
"detector": "mne-icalabel",
"iclabel": {
"label": "eye blink",
"probability": 0.971
}
},
{
"index": 3,
"label": "muscle artifact",
"score": 0.964,
"reason": "ICLabel classified it muscle artifact with probability 0.96 (threshold 0.8)",
"detector": "mne-icalabel",
"iclabel": {
"label": "muscle artifact",
"probability": 0.964
}
},
{
"index": 27,
"label": "muscle artifact",
"score": 0.944,
"reason": "ICLabel classified it muscle artifact with probability 0.94 (threshold 0.8)",
"detector": "mne-icalabel",
"iclabel": {
"label": "muscle artifact",
"probability": 0.944
}
},
{
"index": 10,
"label": "eye blink",
"score": 0.82,
"reason": "ICLabel classified it eye blink with probability 0.82 (threshold 0.8)",
"detector": "mne-icalabel",
"iclabel": {
"label": "eye blink",
"probability": 0.82
}
},
{
"index": 0,
"label": "eye",
"score": 0.664,
"reason": "|correlation| with HEOG_left, HEOG_right, VEOG_lower = 0.66, above the configured threshold 0.5 (measure: correlation)",
"detector": "ICA.find_bads_eog(EOG channels in the recording)",
"iclabel": {
"label": "eye blink",
"probability": 0.986
}
}
],
"n_excluded": 5,
"kept_by_max_remove": [],
"applied": true,
"rank_before": 29,
"rank_after": 24,
"eog": {
"enabled": true,
"channels": [
"HEOG_left",
"HEOG_right",
"VEOG_lower"
],
"provenance": "EOG channels in the recording"
},
"iclabel": {
"enabled": true,
"labels": [
"eye blink",
"other",
"brain",
"muscle artifact",
"eye blink",
"brain",
"other",
"brain",
"eye blink",
"brain",
"eye blink",
"muscle artifact",
"eye blink",
"brain",
"brain",
"brain",
"brain",
"brain",
"brain",
"other",
"brain",
"other",
"other",
"brain",
"brain",
"other",
"other",
"muscle artifact",
"brain"
],
"threshold": 0.8,
"requirements": "extended infomax, 1-100 Hz, average reference (mne-icalabel's stated conditions)"
},
"warning": "ICA did not converge in 500 iterations; the components are whatever the algorithm had reached (logged, not raised)"
},
"status": "ok",
"error": null
},
{
"name": "epoch",
"params": {
"enabled": true,
"source": "annotations",
"stim_channel": null,
"conditions": {
"target": [
11,
22,
33,
44,
55
],
"standard": [
12,
13,
14,
15,
21,
23,
24,
25,
31,
32,
34,
35,
41,
42,
43,
45,
51,
52,
53,
54
]
},
"tmin": -0.2,
"tmax": 0.8,
"baseline": [
-0.2,
0.0
],
"decim": 1,
"detrend": null,
"reject_by_annotation": true,
"event_repeated": "error",
"on_missing": "warn",
"shift_s": 0.0
},
"duration_s": 0.0366,
"notes": {
"source": "annotations",
"events_in_recording": 402,
"event_codes_present": {
"11": 1,
"12": 2,
"13": 3,
"14": 4,
"15": 5,
"201": 6,
"202": 7,
"21": 8,
"22": 9,
"23": 10,
"24": 11,
"25": 12,
"31": 13,
"32": 14,
"33": 15,
"34": 16,
"35": 17,
"41": 18,
"42": 19,
"43": 20,
"44": 21,
"45": 22,
"51": 23,
"52": 24,
"53": 25,
"54": 26,
"55": 27
},
"conditions": {
"target": {
"requested": [
11,
22,
33,
44,
55
],
"codes": [
1,
9,
15,
21,
27
],
"not_found": [],
"n_events": 40
},
"standard": {
"requested": [
12,
13,
14,
15,
21,
23,
24,
25,
31,
32,
34,
35,
41,
42,
43,
45,
51,
52,
53,
54
],
"codes": [
2,
3,
4,
5,
8,
10,
11,
12,
13,
14,
16,
17,
18,
19,
20,
22,
23,
24,
25,
26
],
"not_found": [],
"n_events": 160
}
},
"n_epochs": 200,
"n_epochs_per_condition": {
"target": 40,
"standard": 160
},
"dropped_at_epoching": {},
"tmin_s": -0.2,
"tmax_s": 0.8,
"baseline_s": [
-0.2,
0.0
],
"decim": 1,
"sfreq_hz": 256.0,
"trigger_shift_s": 0.0,
"note": "epoching came after resampling and after every continuous-domain step (L2.8); epochs overlapping a BAD_ annotation were dropped here, before any rejection criterion was applied"
},
"status": "ok",
"error": null
},
{
"name": "reject",
"params": {
"method": "threshold",
"ptp_uv": 100.0,
"flat_uv": 1.0,
"eog_ptp_uv": null,
"max_rejected_fraction": 0.4,
"condition_balance_pp": 10.0,
"autoreject": {
"n_interpolate": [
1,
4
],
"consensus": [],
"cv": 4,
"n_jobs": 1
}
},
"duration_s": 0.0304,
"notes": {
"method": "threshold",
"n_epochs_before": 200,
"criterion": {
"peak_to_peak_uv": 100.0,
"flat_uv": 1.0,
"eog_peak_to_peak_uv": null
},
"criterion_note": "one criterion for every condition, set before the data were looked at (pf-condition-biased-rejection)",
"per_condition": {
"target": {
"n_before": 40,
"n_kept": 38,
"n_rejected": 2,
"percent_rejected": 5.0
},
"standard": {
"n_before": 160,
"n_kept": 153,
"n_rejected": 7,
"percent_rejected": 4.38
}
},
"n_epochs_after": 191,
"percent_rejected": 4.5,
"condition_imbalance_pp": 0.62,
"drop_reasons": {
"CPz": 2,
"Fp1": 1,
"P7": 2,
"PO7": 2,
"P8": 3,
"PO8": 7,
"C3": 1,
"P4": 1,
"Fp2": 1,
"PO3": 1,
"Oz": 1
},
"flags": []
},
"status": "ok",
"error": null
}
],
"config": {
"name": "erpcore-p3",
"description": "Capstone C2 on ERP CORE P3: raw -> cleaned continuous data + epochs + a QC report per subject, in the canonical order of lesson L2.8, driven entirely by this file.\n",
"seed": 20260917,
"dataset": {
"id": "ds-erpcore",
"paradigm": "P3",
"runs": [],
"path": null,
"cache_dir": null,
"synthetic": {}
},
"subjects": {
"include": [
"sub-001",
"sub-002",
"sub-003",
"sub-004",
"sub-005",
"sub-006",
"sub-007",
"sub-008",
"sub-009",
"sub-010"
],
"exclude": {},
"limit": null
},
"output": {
"dir": ".../T/nb-c2-dfglqahu",
"overwrite": true,
"write_raw": true,
"write_epochs": true,
"write_log": true,
"write_qc": true,
"figures": true
},
"versions": {
"on_mismatch": "warn",
"pins": {
"mne": "1.10.2",
"numpy": ">=2.1,<3",
"scipy": ">=1.15,<2"
}
},
"steps": {
"load": {
"preload": true,
"eog_channels": [
"VEOG",
"HEOG"
],
"misc_channels": [],
"rename": {
"FP1": "Fp1",
"FP2": "Fp2"
},
"crop_s": null,
"bad_segments_s": []
},
"montage": {
"name": "standard_1005",
"match_case": true,
"on_missing": "warn"
},
"bad_channels": {
"enabled": true,
"window_s": 5.0,
"flat_uv": 1.0,
"flat_fraction": 0.5,
"deviation_z": 5.0,
"correlation_threshold": 0.4,
"correlation_fraction": 0.3,
"hf_noise_z": 5.0,
"manual": [],
"keep_existing": true,
"max_bad_fraction": 0.2
},
"filter": {
"l_freq": 0.1,
"h_freq": 30.0,
"method": "fir",
"phase": "zero",
"fir_design": "firwin",
"fir_window": "hamming",
"l_trans_bandwidth": "auto",
"h_trans_bandwidth": "auto",
"iir_params": {},
"notch_freqs": [],
"notch_widths": null,
"resample_hz": 256.0,
"ica_l_freq": 1.0,
"ica_h_freq": null,
"justification": {
"highpass": "0.1 Hz: high enough to remove the drift of a 405-s recording, low enough not to distort the P3's late positivity (L2.4, pf-hp-cutoff-erp).\n",
"lowpass": "30 Hz: the P3 is a slow component, and 30 Hz keeps the 60 Hz mains and the muscle band out without touching it. No notch is needed behind a 30 Hz low-pass.\n",
"resample": "1024 Hz costs time and disk for no gain at these cutoffs; 256 Hz is still eight times the low-pass. Resampling happens before epoching so events are converted once.\n",
"ica_highpass": "1 Hz for the ICA fit only: drift dominates the variance and costs components; the unmixing is applied to the 0.1 Hz analysis data (L2.4, L2.6).\n"
}
},
"interpolate": {
"enabled": true,
"method": "spline",
"reset_bads": true,
"exclude": [],
"max_interpolate": 6
},
"reference": {
"type": "average",
"channels": [],
"projection": false,
"add_channels": []
},
"ica": {
"enabled": true,
"method": "infomax",
"fit_params": {
"extended": true
},
"n_components": "rank",
"max_iter": 500,
"decim": 3,
"seed": null,
"reject_by_annotation": true,
"fit_on": "ica_copy",
"eog": {
"enabled": true,
"channels": [],
"measure": "correlation",
"threshold": 0.5
},
"muscle": {
"enabled": false,
"threshold": 0.8
},
"iclabel": {
"enabled": true,
"labels": [
"eye blink",
"muscle artifact",
"heart beat",
"line noise",
"channel noise"
],
"threshold": 0.8
},
"max_remove": 6,
"apply": true
},
"epoch": {
"enabled": true,
"source": "annotations",
"stim_channel": null,
"conditions": {
"target": [
11,
22,
33,
44,
55
],
"standard": [
12,
13,
14,
15,
21,
23,
24,
25,
31,
32,
34,
35,
41,
42,
43,
45,
51,
52,
53,
54
]
},
"tmin": -0.2,
"tmax": 0.8,
"baseline": [
-0.2,
0.0
],
"decim": 1,
"detrend": null,
"reject_by_annotation": true,
"event_repeated": "error",
"on_missing": "warn",
"shift_s": 0.0
},
"reject": {
"method": "threshold",
"ptp_uv": 100.0,
"flat_uv": 1.0,
"eog_ptp_uv": null,
"max_rejected_fraction": 0.4,
"condition_balance_pp": 10.0,
"autoreject": {
"n_interpolate": [
1,
4
],
"consensus": [],
"cv": 4,
"n_jobs": 1
}
}
}
}
}Provenance, versions and seed
| dataset fact | value |
|---|---|
| name | ERP CORE (Compendium of Open Resources and Experiments) |
| license | CC BY 4.0 (OSF node thsqg; TODO(confirm): the component's own LICENSE file says CC BY-SA 4.0 and its dataset_description.json says CC0 -- the author reconciles this, spec section 10.11 item 8) |
| source | https://osf.io/thsqg/ |
| source_doi | TODO(confirm) |
| citation | Kappenman, Farrens, Zhang, Stewart & Luck (2021), ERP CORE |
| sfreq_hz | 1024.00 |
| reference | CMS (Biosemi ActiveTwo online reference) |
| hardware_filters | none recorded (BIDS eeg.json SoftwareFilters: n/a) |
| mains_hz | 60 |
| files | sub-001_task-P3_events.tsv, sub-001_task-P3_eeg.json, sub-001_task-P3_channels.tsv, sub-001_task-P3_electrodes.tsv, sub-001_task-P3_coordsystem.json, sub-001_task-P3_eeg.set, sub-001_task-P3_eeg.fdt |
Seed 20260917, configuration hash 06fd3f5c3717.
| package | in use | pinned in the config |
|---|---|---|
| python | 3.13.5 | — |
| platform | Darwin 25.3.0 (arm64) | — |
| mne | 1.10.2 | 1.10.2 |
| numpy | 2.1.3 | >=2.1,<3 |
| scipy | 1.15.3 | >=1.15,<2 |
| matplotlib | 3.10.6 | — |
| scikit-learn | 1.6.1 | — |
| autoreject | 0.5.0 | — |
| mne-icalabel | 0.9.0 | — |
| python-picard | not installed | — |
| PyYAML | 6.0.2 | — |
The resolved configuration (YAML) — this is what reproduces the run
name: erpcore-p3
description: 'Capstone C2 on ERP CORE P3: raw -> cleaned continuous data + epochs
+ a QC report per subject, in the canonical order of lesson L2.8, driven entirely
by this file.
'
seed: 20260917
dataset:
id: ds-erpcore
paradigm: P3
runs: []
path: null
cache_dir: null
synthetic: {}
subjects:
include:
- sub-001
- sub-002
- sub-003
- sub-004
- sub-005
- sub-006
- sub-007
- sub-008
- sub-009
- sub-010
exclude: {}
limit: null
output:
dir: .../T/nb-c2-dfglqahu
overwrite: true
write_raw: true
write_epochs: true
write_log: true
write_qc: true
figures: true
versions:
on_mismatch: warn
pins:
mne: 1.10.2
numpy: '>=2.1,<3'
scipy: '>=1.15,<2'
steps:
load:
preload: true
eog_channels:
- VEOG
- HEOG
misc_channels: []
rename:
FP1: Fp1
FP2: Fp2
crop_s: null
bad_segments_s: []
montage:
name: standard_1005
match_case: true
on_missing: warn
bad_channels:
enabled: true
window_s: 5.0
flat_uv: 1.0
flat_fraction: 0.5
deviation_z: 5.0
correlation_threshold: 0.4
correlation_fraction: 0.3
hf_noise_z: 5.0
manual: []
keep_existing: true
max_bad_fraction: 0.2
filter:
l_freq: 0.1
h_freq: 30.0
method: fir
phase: zero
fir_design: firwin
fir_window: hamming
l_trans_bandwidth: auto
h_trans_bandwidth: auto
iir_params: {}
notch_freqs: []
notch_widths: null
resample_hz: 256.0
ica_l_freq: 1.0
ica_h_freq: null
justification:
highpass: '0.1 Hz: high enough to remove the drift of a 405-s recording, low
enough not to distort the P3''s late positivity (L2.4, pf-hp-cutoff-erp).
'
lowpass: '30 Hz: the P3 is a slow component, and 30 Hz keeps the 60 Hz mains
and the muscle band out without touching it. No notch is needed behind a 30
Hz low-pass.
'
resample: '1024 Hz costs time and disk for no gain at these cutoffs; 256 Hz
is still eight times the low-pass. Resampling happens before epoching so events
are converted once.
'
ica_highpass: '1 Hz for the ICA fit only: drift dominates the variance and costs
components; the unmixing is applied to the 0.1 Hz analysis data (L2.4, L2.6).
'
interpolate:
enabled: true
method: spline
reset_bads: true
exclude: []
max_interpolate: 6
reference:
type: average
channels: []
projection: false
add_channels: []
ica:
enabled: true
method: infomax
fit_params:
extended: true
n_components: rank
max_iter: 500
decim: 3
seed: null
reject_by_annotation: true
fit_on: ica_copy
eog:
enabled: true
channels: []
measure: correlation
threshold: 0.5
muscle:
enabled: false
threshold: 0.8
iclabel:
enabled: true
labels:
- eye blink
- muscle artifact
- heart beat
- line noise
- channel noise
threshold: 0.8
max_remove: 6
apply: true
epoch:
enabled: true
source: annotations
stim_channel: null
conditions:
target:
- 11
- 22
- 33
- 44
- 55
standard:
- 12
- 13
- 14
- 15
- 21
- 23
- 24
- 25
- 31
- 32
- 34
- 35
- 41
- 42
- 43
- 45
- 51
- 52
- 53
- 54
tmin: -0.2
tmax: 0.8
baseline:
- -0.2
- 0.0
decim: 1
detrend: null
reject_by_annotation: true
event_repeated: error
on_missing: warn
shift_s: 0.0
reject:
method: threshold
ptp_uv: 100.0
flat_uv: 1.0
eog_ptp_uv: null
max_rejected_fraction: 0.4
condition_balance_pp: 10.0
autoreject:
n_interpolate:
- 1
- 4
consensus: []
cv: 4
n_jobs: 1
Generated by eegpipe (pipelines/eegpipe) for the course capstone C2.
Cite a run by its configuration name and hash, the seed and the package versions above.
7. The methods paragraph¶
A reader should be able to follow this and reproduce the pipeline. Every number in it comes from the run logs above, not from memory.
In [9]:
r0 = R[SHOW]
fa = (r0["filter"].get("analysis") or r0["filter"])
ic = r0["filter"].get("ica_copy") or {}
versions = {k: v for k, v in (r0["versions"] or {}).items() if v not in (None, "not installed")}
just = r0["filter"].get("justification") or {}
METHODS = f"""\
Data were the P3 (active visual oddball) paradigm of ERP CORE, {len(SUBJECTS)} of the 40 participants \
({SUBJECTS[0]}-{SUBJECTS[-1]}); 30 EEG and 3 EOG channels recorded with a Biosemi ActiveTwo against the CMS \
arrangement at 1024 Hz with no software filters and 60 Hz mains. The per-subject folders are \
BIDS-compatible and were read as BIDS ({'confirmed with read_raw_bids' if BIDS_OK else 'sidecars read alongside the EEGLAB files'}). \
Processing was scripted with a single configuration file (hash {r0['config_hash']}), executed identically \
for every participant with one random seed ({r0['seed']}) passed into every stochastic step, in the \
canonical order load, montage, bad-channel detection, filter, interpolate, re-reference, ICA, epoch, reject.
Channel names were mapped onto the {'standard_1005' if EEGPIPE is None else config.steps.montage.name} \
template montage (the file spells the frontal pair FP1/FP2, the montage Fp1/Fp2) and the three EOG channels \
were typed as EOG so that they could score ICA components without joining the average reference. Bad \
channels were detected from flatness, robust amplitude deviation, neighbour correlation and \
high-frequency noise on data that had not yet been low-passed, because high-frequency noise is one of the \
criteria; the rule interpolated a median of {np.median(n_bads):.0f} channels \
(range {min(n_bads)}-{max(n_bads)} of 30) by spherical splines.
The continuous data were filtered {fa.get('l_freq_hz')}-{fa.get('h_freq_hz')} Hz \
({str(fa.get('method', 'fir')).upper()}, {fa.get('phase', 'zero')}-phase, \
{fa.get('filter_length_samples', fa.get('fir_length_samples', '?'))} taps = \
{fa.get('filter_length_s', fa.get('fir_length_s', float('nan'))):.1f} s) before epoching, so that the \
filter's edge artifacts lie at the ends of the recording rather than inside every epoch, and resampled to \
{r0['filter'].get('resampled_to_hz', 256)} Hz after the low-pass. \
{' '.join(str(just.get('highpass', '')).split())} {' '.join(str(just.get('lowpass', '')).split())} \
A separate copy high-passed at {ic.get('l_freq_hz', 1.0)} Hz was kept for the ICA fit alone. \
{' '.join(str(just.get('ica_highpass', '')).split())} \
Data were then re-referenced to the average of the 30 EEG channels, with the rank tracked through \
interpolation and referencing ({r0['rank_why']}) and passed to ICA as its component count.
ICA used extended Infomax with the tracked rank and the run's seed. Components were removed when the \
recording's own EOG channels identified them, or when ICLabel classified them into one of the artifact \
classes above the configured probability, subject to a cap; a median of {np.median(n_removed):.0f} \
components was removed (range {min(n_removed)}-{max(n_removed)}), and every removed component is listed in \
the subject's QC report with its class, its score and the sentence explaining why. Labels are algorithmic \
and no person has reviewed them.
Epochs ran from -0.2 to 0.8 s around each stimulus with a -0.2 to 0 s baseline; targets and standards were \
identified from the dataset's own event dictionary, in which a stimulus code with equal digits is a target. \
Epochs were rejected on a peak-to-peak criterion identical for both conditions and fixed before the data \
were seen ({r0['criterion']}), which removed a median of \
{np.median([r['per_condition'].get('target', np.nan) for r in OK]):.1f}% of target and \
{np.median([r['per_condition'].get('standard', np.nan) for r in OK]):.1f}% of standard trials, a median \
per-condition imbalance of {np.nanmedian(gaps):+.1f} percentage points. Subjects were excluded when a \
condition lost more than {MAX_REJECTED_PCT:.0f}% of its trials or kept fewer than \
{MIN_TRIALS_PER_CONDITION}: {', '.join(r['subject'] for r in R.values() if r['excluded']) or 'no subject'} \
met that rule, leaving {len(KEPT)} of {len(SUBJECTS)}.
The P3 was measured as the mean amplitude of the target-minus-standard difference wave at {CH} over \
{WINDOW[0] * 1000:.0f}-{WINDOW[1] * 1000:.0f} ms, a window fixed before any waveform was inspected. \
Package versions are recorded in every run log: {', '.join(f'{k} {v}' for k, v in versions.items())}."""
print(METHODS)
8. The numbers, and the rubric¶
In [10]:
print("nb-c2-clean-pipeline -- Capstone C2 deliverables (draft; TODO(confirm) at author review)")
print(f"Implementation: " + (f"pipelines/eegpipe {EEGPIPE.__version__}, configuration "
f"{CONFIG_PATH.relative_to(_pipelines.parent)}" if EEGPIPE is not None
else "helpers_l2.run_subject (the documented interface)"))
print(f"Dataset: ds-erpcore P3 (CC BY-SA 4.0 per data/directory.yaml; contested at source -- the LICENSE file "
f"says CC BY-SA 4.0, dataset_description.json says CC0, the OSF node record says CC BY 4.0; spec 10.7 "
f"makes the most restrictive reading govern, so share-alike binds what you derive here).")
print(f"Cohort: {'FULL (all participants)' if FULL_COHORT else 'documented subset sub-001..sub-010'} -> "
f"{len(SUBJECTS)} subjects; FULL_COHORT switch at the top of section 1.")
print(f"Configuration hash {R[SHOW]['config_hash']}, seed {R[SHOW]['seed']}.")
print(f"BIDS entry point: {bids_note}")
print(f"Wall clock: {COHORT_S:.0f} s for {len(SUBJECTS)} subjects "
f"({COHORT_S / max(len(SUBJECTS), 1):.0f} s each).")
print()
print("Group summary table (deliverable 4):")
print(l2.fmt_table(group, list(group[0]), floatfmt="{:.2f}"))
print()
print("Per-subject step durations (s):")
print(l2.fmt_table([dict(subject=r["subject"], **{s["name"]: s["duration_s"] for s in r["steps"]},
total=r["duration_s"]) for r in OK], floatfmt="{:.1f}"))
print()
print("ICA components removed, per subject, with the reason recorded for each (deliverable 3):")
for r in OK:
print(f" {r['subject']}: rank {r['rank']} -> {r['rank_after']} after cleaning")
for e in r["removed"]:
print(f" IC{e['component']:<3} {str(e['class']):16s} score {e['probability']:.3f} "
f"{str(e['evidence'])[:92]}")
if not r["removed"]:
print(" nothing met the policy")
print()
print("Rubric, with the measured state of each item:")
RUBRIC = [
("Runs unattended on all subjects -- one call, no prompts, a failure does not stop the cohort",
f"{len(OK)}/{len(SUBJECTS)} completed in one call to run_cohort; "
f"{len(SUBJECTS) - len(OK)} failure(s) recorded as rows; nb-2-8-pipeline section 6 shows the failure "
"path on a subject that does not exist"),
("No condition-biased rejection -- criterion set condition-blind, per-condition table for every subject",
f"one peak-to-peak criterion for both conditions, fixed in the configuration file; imbalance median "
f"{np.nanmedian(gaps):+.1f} pp, range {np.nanmin(gaps):+.1f} to {np.nanmax(gaps):+.1f} pp; "
f"{len([r for r in OK if abs(r['imbalance_pp']) >= 20])} subject(s) at or above 20 pp"),
("QC report readable by a stranger",
f"{len([f for f in files if f.suffix == '.html'])} self-contained HTML pages plus a cohort index; "
"one is rendered in section 6"),
("Driven entirely by a configuration file, seed and versions in the run log",
f"hash {R[SHOW]['config_hash']}, seed {R[SHOW]['seed']}, versions recorded per subject; the two "
"overrides applied here are arguments (output directory, ICLabel on), not edits"),
("Canonical order implemented and justified",
("load -> montage -> bad-channel detection -> filter -> interpolate -> re-reference -> ICA -> epoch -> "
"reject") + "; justified step by step in the methods paragraph, and the filter's justification is "
"carried in the configuration file itself"),
("Rank tracked through interpolation and re-referencing and passed to ICA",
f"e.g. {SHOW}: {R[SHOW]['rank_why']}; ICA component count = that number"),
("Every removed ICA component has a class and evidence; the number removed is comparable across subjects",
f"median {np.median(n_removed):.0f}, range {min(n_removed)}-{max(n_removed)}; each carries a detector, "
"a score and a sentence"),
("The subject-exclusion rule was fixed in advance and applied blind to the effect",
f"a condition may lose at most {MAX_REJECTED_PCT:.0f} % of its trials (the configuration's "
f"max_rejected_fraction) and must keep at least {MIN_TRIALS_PER_CONDITION}: "
+ (", ".join(f"{r['subject']} ({r['exclusion_reason']})" for r in R.values() if r['excluded'])
or "no subject crossed it") + f"; {len(KEPT)} of {len(SUBJECTS)} retained, and the rule was applied "
"before any grand average was computed"),
("Dataset-specific documented defects handled explicitly",
"ds-erpcore documents no defective subjects; the configuration's subjects.exclude field is where "
"ds-eegbci's S088/S089/S092/S100 and S038/S104 go, each with its reason (pipelines/configs/eegbci.yaml)"),
]
for item, state in RUBRIC:
print(f" [x] {item}\n {state}")
print()
print(f"Deliverables written: {len([f for f in files if f.name.endswith('_raw.fif')])} cleaned continuous "
f"recordings, {len([f for f in files if f.name.endswith('_epo.fif')])} epoch files, "
f"{len([f for f in files if f.suffix == '.html'])} QC pages and "
f"{len([f for f in files if f.name == 'run-log.json'])} run logs, "
f"{sum(f.stat().st_size for f in files) / 1e6:.0f} MB. Set EEG_COURSE_C2_OUT to keep them.")
if amps:
v = np.array([a[f"{CH} mean (uV)"] for a in amps])
print()
print(f"Grand-average P3 across the {len(v)} retained subjects: {CH} mean amplitude {v.mean():+.2f} uV "
f"(SD {v.std(ddof=1):.2f}, SEM {v.std(ddof=1) / np.sqrt(len(v)):.2f}). "
"C3 reuses this pipeline unchanged.")
_out.cleanup()
print("\ntemporary output directory removed; the ERP CORE downloads stay in the cache.")