nb-2-1-bids · Loading data and BIDS (L2.1)¶
Lesson L2.1 · Level 2 · Status draft — for expert review; uncertain points carry TODO(confirm).
Four steps, in the order the lesson puts them:
- Load and read the file, not the documentation. One subject of
ds-eegbcistraight from its EDF, before anything is renamed: channel names, channel types, header filter fields, annotations, event counts and intervals. - BIDS-ify a messy folder. The exercise's messy folder is built here from that subject (arbitrary file names, no sidecars, a free-text notes file), and then converted with
mne-bids— which forces you to establish the line frequency, the reference, the channel types and the units before it will write. - Read it back with
read_raw_bidsand check the round trip: names, types, sampling rate, event counts. - Compare with somebody else's conversion. The same subject exists in the official BIDS mirror of this dataset (OpenNeuro ds004362, CC0). Only the sidecars are downloaded (a few kB); the comparison of the two
_events.tsvfiles, the two_channels.tsvfiles and the two_eeg.jsonsidecars is the checkable step of this lesson.
Data
ds-eegbci(EEGMMIDB; PhysioNet DOI 10.13026/C28G6P; ODC-By 1.0): subject S001, runs R01 (eyes open, ~1 min), R02 (eyes closed, ~1 min) and R03 (motor execution, ~2 min). 64 channels, 160 Hz, no hardware filters, EDF+. About 5 MB of downloads.- The official BIDS mirror of the same dataset: OpenNeuro ds004362 (CC0), read through its public S3 bucket — sidecars only, nothing heavy.
The documented defects of ds-eegbci (subjects S088, S089, S092, S100 with inconsistent event timestamps; S038 and S104 also commonly dropped; Sharbrough-style channel labels) are handled explicitly in section 1 and printed in the final cell.
# Setup: dependencies, the shared helpers, non-interactive plotting.
import importlib.util
import subprocess
import sys
import warnings
from pathlib import Path
# 1. Dependencies are pinned in notebooks/requirements.txt. Nothing is installed
# when the pinned stack is already present (local runs, CI); a fresh Colab or
# Binder kernel installs it once. On Colab, run from a clone of the repository
# so that notebooks/_shared/ is available (repository URL: TODO(confirm), spec
# section 13 item 3).
_needed = ("mne", "mne_bids", "scipy", "matplotlib", "pooch")
_missing = [p for p in _needed if importlib.util.find_spec(p) is None]
if _missing:
_req = next((d / "requirements.txt" for d in (Path.cwd(), *Path.cwd().parents)
if (d / "requirements.txt").exists()), None)
_cmd = [sys.executable, "-m", "pip", "install", "-q"]
_cmd += ["-r", str(_req)] if _req else ["mne==1.10.2", "pooch>=1.8"]
if "mne_bids" in _missing:
_cmd += ["mne-bids==0.19.0"]
subprocess.check_call(_cmd)
# 2. Shared helpers (notebooks/_shared/helpers.py and helpers_l2.py), located
# relative to the working directory -- notebooks/<level>/ or notebooks/ --
# never through an absolute path.
_shared = next((d / "_shared" for d in (Path.cwd(), *Path.cwd().parents)
if (d / "_shared" / "helpers_l2.py").exists()), None)
if _shared is None:
raise FileNotFoundError("start the kernel in notebooks/L2/ (or notebooks/) so that _shared/helpers_l2.py is found")
sys.path.insert(0, str(_shared))
import helpers
import helpers_l2 as l2
# 3. Plotting: Jupyter's default inline backend renders static PNGs through Agg
# (no windows, nothing blocks); outside Jupyter the helpers select Agg. Every
# MNE figure is requested with show=False, and plt.show() renders each cell's
# figures in place.
import matplotlib.pyplot as plt
import numpy as np
import mne
import mne_bids
# Warnings are worth reading, so they are not silenced -- but their default format prints the
# absolute path of the file that raised them, which is nobody else's business and would put this
# machine's directory layout into the saved outputs. Only the class and the message are shown.
warnings.formatwarning = lambda message, category, *a, **k: f"{category.__name__}: {message}\n"
mne.viz.set_browser_backend("matplotlib", verbose=False)
mne.set_log_level("WARNING")
plt.rcParams["figure.dpi"] = 72
print(f"MNE {mne.__version__}, mne-bids {mne_bids.__version__}; helpers imported from notebooks/_shared; "
"downloads go to MNE's default data directory unless EEG_COURSE_DATA is set")
1. Read the file as it is stored¶
helpers.load_spine standardises this dataset for you (it strips the trailing dots from the Sharbrough labels, attaches standard_1005 and refuses the documented defective subjects). That is the right default for every other notebook and exactly the wrong thing here, because the point of this lesson is what the file actually contains. So the EDF path is resolved through MNE's own downloader and read directly.
from mne.datasets import eegbci
SUBJECT, RUNS = 1, [1, 2, 3] # S001, R01 eyes open / R02 eyes closed / R03 motor execution
RUN_MEANING = {1: helpers.EEGBCI_RUNS["R01"], 2: helpers.EEGBCI_RUNS["R02"], 3: helpers.EEGBCI_RUNS["R03"]}
with warnings.catch_warnings():
warnings.simplefilter("ignore")
edf_paths = eegbci.load_data(SUBJECT, RUNS, update_path=False, verbose=False)
print(f"{len(edf_paths)} EDF files for S{SUBJECT:03d}, runs {RUNS}: "
+ ", ".join(Path(p).name for p in edf_paths))
print(f"excluded by the catalog and refused by helpers.load_spine: {', '.join(helpers.EEGBCI_EXCLUDE_DEFAULT)} "
f"(inconsistent/overlapping event timestamps); {', '.join(helpers.EEGBCI_EXCLUDE_OPTIONAL)} "
"(several reports additionally drop them). S001 is in neither list.")
raws_native = []
for p, r in zip(edf_paths, RUNS):
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
raw = mne.io.read_raw_edf(p, preload=True, verbose=False)
raws_native.append(raw)
print(f"\nR{r:02d} -- {RUN_MEANING[r]}")
print(f" sfreq {raw.info['sfreq']:g} Hz | {len(raw.ch_names)} channels | {raw.times[-1] + 1 / raw.info['sfreq']:.1f} s "
f"| header highpass {raw.info['highpass']:g} Hz, lowpass {raw.info['lowpass']:g} Hz")
print(f" channel types: {sorted(set(raw.get_channel_types()))}")
print(f" first six names as stored: {raw.ch_names[:6]}")
print(f" annotations: {len(raw.annotations)} -> {sorted(set(raw.annotations.description))}")
for c in caught:
print(" reader warns:", str(c.message)[:120])
Three readings from that output, and all three are decisions waiting to be made.
The names carry punctuation. Fc5., Fp1., Af7. are Sharbrough-style labels with trailing dots; no standard montage knows them, and set_montage with the default on_missing='warn' will attach nothing and tell you so in a warning you will not read in a loop over 100 subjects. The cell below shows both outcomes.
Every channel is typed eeg. EDF does not record channel types, so MNE assigns the default. This dataset genuinely has 64 EEG channels and nothing else, so the default happens to be right — but "happens to be right" is not the same as "was checked", and on a file with an EOG or a trigger line it would be wrong (L2.1).
The header's filter fields read 0 Hz / Nyquist. That is this dataset telling the truth: it was recorded with no hardware filters, which is why the site uses it for drift and line noise (L1.6, L2.4).
montage = mne.channels.make_standard_montage("standard_1005")
raw = raws_native[2].copy() # R03: the run with real events
try:
raw.copy().set_montage(montage, on_missing="raise")
except ValueError as e:
print("set_montage(on_missing='raise') before renaming:", str(e)[:160], "...")
eegbci.standardize(raw) # strip the trailing dots, re-case to 10-10 names
print(f"\nafter eegbci.standardize: {raw.ch_names[:6]} ...")
raw.set_montage(montage, on_missing="raise")
pos = raw.get_montage().get_positions()["ch_pos"]
print(f"set_montage(on_missing='raise') passes: {len(pos)} of {len(raw.ch_names)} channels have a position")
raw.set_channel_types({c: "eeg" for c in raw.ch_names}) # the decision, made explicitly
print(f"channel types after the explicit decision: {sorted(set(raw.get_channel_types()))}")
events, event_id = mne.events_from_annotations(raw, verbose=False)
onsets = events[:, 0] / raw.info["sfreq"]
print(f"\nR03 events: {len(events)} in {len(event_id)} classes {event_id}")
for name, code in event_id.items():
print(f" {name}: {(events[:, 2] == code).sum():3d} events")
print(f" first onset {onsets[0]:.2f} s, last {onsets[-1]:.2f} s of {raw.times[-1]:.1f} s; "
f"inter-event intervals: median {np.median(np.diff(onsets)):.2f} s, "
f"min {np.diff(onsets).min():.2f} s, max {np.diff(onsets).max():.2f} s")
fig, axes = plt.subplots(1, 2, figsize=(13, 4.2), gridspec_kw=dict(width_ratios=[1, 1.6]))
raw.plot_sensors(show_names=["Fp1", "Fz", "Cz", "Pz", "Oz", "C3", "C4"], axes=axes[0], show=False)
axes[0].set_title("S001 after renaming: standard_1005 positions (64 of 64 matched)")
helpers.plot_traces(raw, ["Fp1", "F3", "C3", "Cz", "C4", "P3", "Pz", "O1"], t0=10, duration=10,
spacing_uV=100, ax=axes[1], title="S001 R03, 10 s of raw EDF (uV)")
fig.tight_layout()
plt.show() # render the static figure(s) of this cell inline
2. The messy folder¶
The exercise asks you to BIDS-ify a messy folder. This is what one looks like: the same three runs, saved under names that mean something only to the person who made them, with the metadata in a free-text file instead of a sidecar. Nothing here is a strawman — file names like these are what most lab archives actually contain.
The folder is built in a temporary directory so that the notebook leaves nothing behind; a learner working through the exercise can point the same code at their own folder.
import shutil
import tempfile
_tmp = tempfile.TemporaryDirectory(prefix="nb-2-1-")
WORK = Path(_tmp.name)
MESSY, BIDS_ROOT = WORK / "messy_folder", WORK / "bids_out"
MESSY.mkdir(parents=True)
MESSY_NAMES = {1: "subj1_eyesopen_FINAL.edf", 2: "subj1_eyesclosed_FINAL.edf", 3: "subj1_task_run3(copy).edf"}
for p, r in zip(edf_paths, RUNS):
shutil.copyfile(p, MESSY / MESSY_NAMES[r])
(MESSY / "notes about the recordings.txt").write_text(
"S001, 64ch cap, 160Hz. amp had no filters on. mains 60Hz here.\n"
"ref = ear lobe (left or right, ask the RA)\n"
"run3 = open/close fist task, T0 rest T1 left T2 right\n"
"units are microvolts I think\n", encoding="utf-8")
print("the messy folder:")
for f in sorted(MESSY.iterdir()):
print(f" {f.name:34s} {f.stat().st_size / 1e6:6.2f} MB")
print("\nwhat a BIDS writer will ask for that this folder does not answer in machine-readable form:")
for q in ("task name", "SamplingFrequency", "PowerLineFrequency", "EEGReference", "SoftwareFilters",
"channel types (EEG/EOG/ECG/stim/misc)", "channel units", "event onsets in seconds + condition names",
"participant id and any demographics", "dataset name, licence, authors"):
print(f" - {q}")
3. Convert with mne-bids¶
Every argument below is a fact that had to be established first, which is the whole point of the exercise (the lesson's note: converting to BIDS is a forced metadata audit). Where the fact comes from the site's dataset catalog rather than from the file, it is marked in the printed table.
Two mechanical points. write_raw_bids converts formats according to what BIDS allows for EEG; here the preloaded EDF is written as BrainVision, so the file that comes out is not the file that went in, and the conversion is recorded. And the events are already in the Raw as annotations, so they are not passed again through events= — passing both writes them twice and the round trip comes back with double the annotations.
from mne_bids import BIDSPath, write_raw_bids, read_raw_bids, print_dir_tree
TASK = {1: "eyesopen", 2: "eyesclosed", 3: "motorexec"}
LINE_FREQ = helpers.DATASETS["ds-eegbci"]["mains_hz"] # 60 Hz, from the catalog
REFERENCE = "left or right ear lobe (catalog: TODO(confirm) which)" # catalog + the mirror's own sidecar
POWERLINE_SOURCE = "catalog (data/catalog/datasets/eegmmidb.md), not the file"
written = {}
for messy_file, r in [(MESSY / MESSY_NAMES[r], r) for r in RUNS]:
raw_m = mne.io.read_raw_edf(messy_file, preload=True, verbose=False)
eegbci.standardize(raw_m) # names first (L2.1)
raw_m.set_montage(montage, on_missing="raise") # positions second
raw_m.set_channel_types({c: "eeg" for c in raw_m.ch_names}) # types are a decision
raw_m.info["line_freq"] = LINE_FREQ # required by BIDS; not in the file
bp = BIDSPath(subject=f"{SUBJECT:03d}", task=TASK[r], run=f"{r:02d}", datatype="eeg", root=BIDS_ROOT)
with warnings.catch_warnings():
warnings.simplefilter("ignore")
written[r] = write_raw_bids(raw_m, bp, format="BrainVision", allow_preload=True,
overwrite=True, verbose=False)
# mne-bids cannot invent the reference or the software filters; they are written afterwards.
mne_bids.update_sidecar_json(bids_path=written[r].copy().update(extension=".json"),
entries={"EEGReference": REFERENCE, "SoftwareFilters": "n/a",
"HardwareFilters": "n/a (the catalog documents no online filtering "
"and no notch)",
"EEGPlacementScheme": "10-10 (extended 10-20)",
"Manufacturer": "n/a (BCI2000 acquisition; TODO(confirm) the amplifier)",
"InstitutionName": "n/a"},
verbose=False)
print(f"R{r:02d} -> {written[r].basename} ({len(raw_m.ch_names)} ch, {len(raw_m.annotations)} annotations)")
# The two sidecars BIDS asks for that mne-bids cannot fill from the data.
import json
(BIDS_ROOT / "dataset_description.json").write_text(json.dumps({
"Name": "EEG Motor Movement/Imagery Dataset -- three-run subset converted for nb-2-1",
"BIDSVersion": "1.8.0",
"License": helpers.DATASETS["ds-eegbci"]["license"],
"Authors": ["Schalk, G.", "McFarland, D. J.", "Hinterberger, T.", "Birbaumer, N.", "Wolpaw, J. R."],
"ReferencesAndLinks": [helpers.DATASETS["ds-eegbci"]["source"]],
"DatasetDOI": f"doi:{helpers.DATASETS['ds-eegbci']['doi']}",
}, indent=2), encoding="utf-8")
(BIDS_ROOT / "participants.json").write_text(json.dumps({
"participant_id": {"Description": "Unique participant identifier"},
"age": {"Description": "Age in years", "Units": "years"},
"sex": {"Description": "Biological sex", "Levels": {"M": "male", "F": "female", "n/a": "not documented"}},
"handedness": {"Description": "Self-reported handedness"},
}, indent=2), encoding="utf-8")
print("\nwrote dataset_description.json and participants.json by hand: a writer cannot invent a licence.")
for lock in BIDS_ROOT.rglob("*.lock"): # mne-bids leaves file locks behind; they are not BIDS files
lock.unlink()
print_dir_tree(BIDS_ROOT, max_depth=3)
sidecar = json.loads((written[3].copy().update(extension=".json")).fpath.read_text())
rows = [[k, str(v), ("the file" if k in ("SamplingFrequency", "RecordingDuration", "EEGChannelCount",
"RecordingType", "EOGChannelCount", "ECGChannelCount",
"EMGChannelCount", "MiscChannelCount") else "you")]
for k, v in sidecar.items()]
print("the _eeg.json for R03, and where each value came from:\n")
print(l2.fmt_table([{"field": k, "value": v[:60], "source": s} for k, v, s in rows],
["field", "value", "source"]))
print(f"\nPowerLineFrequency came from the {POWERLINE_SOURCE}.")
print(f"EEGReference as recorded here: {REFERENCE!r} -- mne-bids writes 'n/a' unless you set it, "
"and 'n/a' is an honest answer only if you genuinely do not know.")
channels_tsv = (written[3].copy().update(suffix="channels", extension=".tsv")).fpath.read_text().splitlines()
print(f"\n_channels.tsv header: {channels_tsv[0]}")
print(f"first three rows: " + " | ".join(channels_tsv[1:4]))
events_tsv = (written[3].copy().update(suffix="events", extension=".tsv")).fpath.read_text().splitlines()
print(f"\n_events.tsv: {len(events_tsv) - 1} rows, header {events_tsv[0]}")
print("first three rows: " + " | ".join(events_tsv[1:4]))
4. The round trip¶
read_raw_bids should hand back the same recording: the same channel names in the same order, the same types, the same sampling rate and the same events. This is the test that matters, and it is worth running on every conversion — a silent change here becomes an unexplainable group result later.
ROUND_TRIP = {}
for r in RUNS:
with warnings.catch_warnings():
warnings.simplefilter("ignore")
back = read_raw_bids(written[r], verbose=False)
orig = raws_native[RUNS.index(r)].copy()
eegbci.standardize(orig)
ev_back, id_back = mne.events_from_annotations(back, verbose=False)
ev_orig, id_orig = mne.events_from_annotations(orig, verbose=False)
checks = {
"channel names": back.ch_names == orig.ch_names,
"channel types": back.get_channel_types() == orig.get_channel_types(),
"sampling rate": back.info["sfreq"] == orig.info["sfreq"],
"n samples": back.n_times == orig.n_times,
"n events": len(ev_back) == len(ev_orig),
"event classes": set(id_back) == set(id_orig),
"data (max abs difference < 1 nV)": float(np.abs(back.get_data() - orig.get_data()).max()) < 1e-9,
}
ROUND_TRIP[r] = dict(checks, n_events=len(ev_back),
max_abs_diff_uV=float(np.abs(back.get_data() - orig.get_data()).max()) * 1e6)
status = "OK" if all(checks.values()) else "MISMATCH"
print(f"R{r:02d} round trip: {status:8s} {len(ev_back):2d} events, "
f"max |difference| {ROUND_TRIP[r]['max_abs_diff_uV']:.4g} uV, "
+ ", ".join(k for k, v in checks.items() if not v))
The data survive the EDF → BrainVision conversion to well below a nanovolt, which is the numerical precision of the format change and not a fact about EEG. What does not survive automatically is anything the source format never held: the reference, the line frequency, the channel types. Those are in the sidecar because you put them there.
5. Compare with the official mirror¶
The same dataset was converted to BIDS by somebody else and published as OpenNeuro ds004362 (CC0). That makes this lesson checkable: two independent conversions of one recording, and every difference is either a choice or a mistake. Only sidecars are downloaded — a few kilobytes — through OpenNeuro's public S3 bucket.
import urllib.request
MIRROR = "ds004362"
MIRROR_BASE = f"https://s3.amazonaws.com/openneuro.org/{MIRROR}/"
# The mirror files every run under one task name with an unpadded run entity.
MIRROR_FILES = {r: f"sub-001/eeg/sub-001_task-motion_run-{r}_" for r in RUNS}
def fetch_text(name, timeout=30):
"""Fetch one small text file from the mirror; returns None (with a note) if it cannot be read."""
try:
with urllib.request.urlopen(MIRROR_BASE + name, timeout=timeout) as resp:
return resp.read().decode("utf-8")
except Exception as e:
print(f" could not read {name}: {type(e).__name__}: {e}")
return None
mirror = {}
for r in RUNS:
mirror[r] = {kind: fetch_text(MIRROR_FILES[r] + suffix)
for kind, suffix in (("eeg.json", "eeg.json"), ("events.tsv", "events.tsv"),
("channels.tsv", "channels.tsv"))}
mirror_dd = fetch_text("dataset_description.json")
if mirror_dd:
dd = json.loads(mirror_dd)
print(f"{MIRROR}: {dd['Name']!r}, BIDS {dd['BIDSVersion']}, licence {dd['License']}, "
f"DOI {dd.get('DatasetDOI', 'n/a')}")
MIRROR_EVENT_COUNTS, MIRROR_LABELS = {}, {}
for r in RUNS:
lines = [l for l in (mirror[r]["events.tsv"] or "").splitlines()[1:] if l.strip()]
MIRROR_EVENT_COUNTS[r] = len(lines) if mirror[r]["events.tsv"] else None
MIRROR_LABELS[r] = sorted({l.split(" ")[-1] for l in lines})
print(f" R{r:02d}: mirror _events.tsv has {MIRROR_EVENT_COUNTS[r]} rows labelled {MIRROR_LABELS[r]}; "
f"our conversion has {ROUND_TRIP[r]['n_events']} rows labelled "
f"{sorted(set(raws_native[RUNS.index(r)].annotations.description))}")
r = 3 # the run with real events
mine_json = json.loads((written[r].copy().update(extension=".json")).fpath.read_text())
theirs_json = json.loads(mirror[r]["eeg.json"]) if mirror[r]["eeg.json"] else {}
keys = sorted(set(mine_json) | set(theirs_json))
rows = [{"field": k, "ours": str(mine_json.get(k, "-"))[:44], "mirror": str(theirs_json.get(k, "-"))[:44],
"same": "yes" if mine_json.get(k) == theirs_json.get(k) else "NO"} for k in keys]
print(f"_eeg.json for run {r}, ours versus {MIRROR}:\n")
print(l2.fmt_table(rows, ["field", "ours", "mirror", "same"]))
mine_ch = [l.split("\t") for l in (written[r].copy().update(suffix="channels", extension=".tsv")
).fpath.read_text().splitlines()]
their_ch = [l.split("\t") for l in mirror[r]["channels.tsv"].splitlines()] if mirror[r]["channels.tsv"] else []
mine_names = [row[0] for row in mine_ch[1:]]
their_names = [row[0] for row in their_ch[1:]]
print(f"\n_channels.tsv: ours {len(mine_names)} rows {mine_ch[0]}, mirror {len(their_names)} rows {their_ch[0]}")
print(f" same order, case-insensitively: {[n.lower() for n in mine_names] == [n.lower() for n in their_names]}")
print(f" names that differ in spelling: "
f"{[(a, b) for a, b in zip(mine_names, their_names) if a != b][:5]} ... "
f"({sum(a != b for a, b in zip(mine_names, their_names))} of {len(mine_names)})")
their_types = {row[1] for row in their_ch[1:]}
their_units = {row[2] for row in their_ch[1:]}
mine_types = {row[mine_ch[0].index("type")] for row in mine_ch[1:]}
mine_units = {row[mine_ch[0].index("units")] for row in mine_ch[1:]}
print(f" types -- ours {sorted(mine_types)}, mirror {sorted(their_types)}")
print(f" units -- ours {sorted(mine_units)}, mirror {sorted(their_units)}")
This is the finding the exercise is built on, and it is not that one conversion is wrong.
- The event counts agree, which is the substantive check: both conversions found the same number of events in the same run, so neither lost a trial.
- The mirror's
_channels.tsvleavestypeandunitsasn/a. BIDS allows it, and the validator will not complain, but a pipeline that readsunitsto decide whether the numbers are volts or microvolts gets nothing — the factor-of-a-million error the lesson warns about. - The event labels differ: the mirror writes
BASE1T0,TASK1T0,TASK1T1— the run context folded into the label — where a conversion from the annotations writes the bareT0,T1,T2that the EDF stores. Both are faithful; only one of them tells you which run a row came from if the files are ever concatenated. - The mirror's
EEGReferencereads "Left or right ear lobe", which is exactly the free text the lesson describes: it records genuine uncertainty rather than hiding it. - Our sidecar reports
EEGChannelCountand aRecordingDurationread from the data; the mirror's values come from its own converter. Any disagreement here is worth chasing, because it means the two files are not the same recording.
6. The validator's job, and what can be checked here¶
The official BIDS validator is a separate tool and is not installed by this notebook. What can be checked in Python is the structural part of its job — the files that must exist, the entities that must match, the columns that must be present — and that is what the cell below does. Run the real validator on your own conversion as well; it catches naming mistakes this check does not.
def structural_checks(root: Path) -> list[dict]:
"""A subset of what the BIDS validator checks. Not a substitute for it."""
out = []
def add(item, ok, value="", note=""):
out.append({"check": item, "status": "ok" if ok else "FAIL", "value": str(value)[:52], "note": note})
add("dataset_description.json exists", (root / "dataset_description.json").exists())
if (root / "dataset_description.json").exists():
dd = json.loads((root / "dataset_description.json").read_text())
for k in ("Name", "BIDSVersion", "License"):
add(f"dataset_description.{k}", k in dd, dd.get(k, ""))
add("participants.tsv exists", (root / "participants.tsv").exists())
add("participants.json describes the columns", (root / "participants.json").exists())
add("README exists", (root / "README").exists())
subs = sorted(d.name for d in root.iterdir() if d.is_dir() and d.name.startswith("sub-"))
add("at least one sub-<label> directory", bool(subs), ", ".join(subs))
if (root / "participants.tsv").exists():
ids = [l.split("\t")[0] for l in (root / "participants.tsv").read_text().splitlines()[1:] if l.strip()]
add("every subject directory has a participants.tsv row", set(subs) <= set(ids), ", ".join(ids))
for eeg_json in sorted(root.rglob("*_eeg.json")):
stem = eeg_json.name[: -len("_eeg.json")]
j = json.loads(eeg_json.read_text())
missing = [k for k in ("TaskName", "SamplingFrequency", "PowerLineFrequency", "EEGReference",
"SoftwareFilters") if k not in j or j[k] in (None, "")]
vague = [k for k in ("EEGReference",) if str(j.get(k)).strip().lower() == "n/a"]
add(f"{stem}: required _eeg.json fields", not missing,
"missing: " + ", ".join(missing) if missing else "all present")
add(f"{stem}: EEGReference is not a bare 'n/a'", not vague,
"n/a" if vague else str(j.get("EEGReference"))[:44],
"BIDS allows n/a; it is honest only when you genuinely do not know. "
"SoftwareFilters 'n/a' here IS the fact: no software filter was applied, and HardwareFilters "
"records that the amplifier applied none either")
for suffix, cols in (("channels.tsv", ("name", "type", "units")), ("events.tsv", ("onset", "duration"))):
f = eeg_json.with_name(f"{stem}_{suffix}")
header = f.read_text().splitlines()[0].split("\t") if f.exists() else []
add(f"{stem}: {suffix} columns", f.exists() and all(c in header for c in cols), ", ".join(header))
add(f"{stem}: PowerLineFrequency is a number, not n/a",
isinstance(j.get("PowerLineFrequency"), (int, float)), j.get("PowerLineFrequency"))
return out
checks = structural_checks(BIDS_ROOT)
print(l2.fmt_table(checks, ["check", "status", "value"]))
n_fail = sum(c["status"] == "FAIL" for c in checks)
print(f"\n{len(checks) - n_fail} of {len(checks)} structural checks pass; {n_fail} fail.")
print("This is not the BIDS validator. Run it on your own output and read every warning: "
"https://bids-standard.github.io/bids-validator/")
7. The numbers¶
print("nb-2-1-bids -- L2.1 exercise numbers (draft; TODO(confirm) at author review)")
print(f"Data: ds-eegbci S{SUBJECT:03d} runs R01/R02/R03 (PhysioNet DOI {helpers.DATASETS['ds-eegbci']['doi']}; "
f"{helpers.DATASETS['ds-eegbci']['license']}), converted here with mne-bids {mne_bids.__version__}; "
f"official BIDS mirror OpenNeuro {MIRROR} (CC0) read for comparison.")
print()
print("ex-2-1-mirror-diff -- how many events does the mirror's _events.tsv contain for the converted run?")
for r in RUNS:
print(f" run-{r:02d} ({RUN_MEANING[r].split(':')[0]}): mirror {MIRROR_EVENT_COUNTS[r]} events, "
f"our conversion {ROUND_TRIP[r]['n_events']} events "
f"-> {'agree' if MIRROR_EVENT_COUNTS[r] == ROUND_TRIP[r]['n_events'] else 'DISAGREE'}")
print(f" The exercise's run is R03 (the one with real events): answer = {MIRROR_EVENT_COUNTS[3]} events "
"(tolerance 0; a count, and a mismatch with your own conversion is the finding).")
print()
print("Round trip through mne-bids (read_raw_bids against the source EDF):")
for r in RUNS:
rt = ROUND_TRIP[r]
print(f" R{r:02d}: names/types/sfreq/n_samples/events all "
f"{'unchanged' if all(v for k, v in rt.items() if isinstance(v, bool)) else 'CHANGED'}; "
f"max |difference| {rt['max_abs_diff_uV']:.3g} uV (EDF -> BrainVision conversion precision)")
print()
print("ex-2-1-bidsify-checklist -- the state of the conversion made here, item by item:")
CHECKLIST = [
("Channel names re-cased and de-punctuated; set_montage(on_missing='raise') passes",
f"yes -- {len(pos)}/{len(raw.ch_names)} positions attached"),
("Channel types set explicitly; no trigger line typed as EEG",
f"yes -- {sorted(set(raw.get_channel_types()))}; this dataset has no EOG/ECG/stim channel"),
("dataset_description.json with Name, BIDSVersion, License", "yes -- written by hand"),
("participants.tsv one row per subject; participants.json describes every column",
"yes -- mne-bids writes the tsv, the json was written by hand"),
("_eeg.json has SamplingFrequency, PowerLineFrequency, EEGReference, SoftwareFilters",
f"SamplingFrequency from the file; PowerLineFrequency {LINE_FREQ:g} Hz from the catalog; EEGReference "
f"{REFERENCE!r} written with update_sidecar_json; SoftwareFilters 'n/a' (the amplifier's own band-pass "
"is all that happened) -- TODO(confirm) which ear lobe"),
("_channels.tsv lists name, type and units, matching what the data holds",
f"ours {sorted(mine_units)}; the mirror leaves both n/a"),
("_events.tsv onsets in seconds; counts per condition match the protocol",
f"yes -- {len(events_tsv) - 1} rows for R03, same count as the mirror"),
("Subjects with documented defects excluded with the reason recorded, or the defect noted",
f"S088/S089/S092/S100 (inconsistent event timestamps) and S038/S104 refused by helpers.load_spine; "
f"S{SUBJECT:03d} is in neither list"),
("read_raw_bids round-trips names, types, sampling rate and event counts",
"yes -- all three runs"),
("The BIDS validator runs with no errors",
f"NOT CHECKED here -- {len(checks) - n_fail}/{len(checks)} structural checks pass; "
"run the official validator yourself"),
]
for item, state in CHECKLIST:
print(f" [{'x' if not state.startswith('NOT') else ' '}] {item}\n {state}")
print()
print("Differences between the two conversions that are choices, not errors: event labels "
"(mirror 'TASK1T1' vs annotation 'T1'), channel-name case (mirror 'Fc5' vs MNE's 'FC5'), and the "
"mirror's n/a channel types and units.")
_tmp.cleanup()
print("\ntemporary BIDS folders removed; the EDF downloads stay in MNE's data directory.")