Loading data and BIDS: read the file as it is stored, BIDS-ify a messy folder with mne-bids, round-trip it, and compare with the official OpenNeuro mirror

nb-2-1-bids Level 2 · Preprocessing as a Pipeline ~3 min Used in L2.1 · Loading data and BIDS

Downloads from ds-eegbci when you run it.

Download the notebook (.ipynb) Outputs below are the ones stored when it was executed — you do not need to run anything to read it.

nb-2-1-bids · Loading data and BIDS (L2.1)

Lesson L2.1 · Level 2 · Status draft — for expert review; uncertain points carry TODO(confirm).

Four steps, in the order the lesson puts them:

  1. Load and read the file, not the documentation. One subject of ds-eegbci straight from its EDF, before anything is renamed: channel names, channel types, header filter fields, annotations, event counts and intervals.
  2. BIDS-ify a messy folder. The exercise's messy folder is built here from that subject (arbitrary file names, no sidecars, a free-text notes file), and then converted with mne-bids — which forces you to establish the line frequency, the reference, the channel types and the units before it will write.
  3. Read it back with read_raw_bids and check the round trip: names, types, sampling rate, event counts.
  4. Compare with somebody else's conversion. The same subject exists in the official BIDS mirror of this dataset (OpenNeuro ds004362, CC0). Only the sidecars are downloaded (a few kB); the comparison of the two _events.tsv files, the two _channels.tsv files and the two _eeg.json sidecars is the checkable step of this lesson.

Data

  • ds-eegbci (EEGMMIDB; PhysioNet DOI 10.13026/C28G6P; ODC-By 1.0): subject S001, runs R01 (eyes open, ~1 min), R02 (eyes closed, ~1 min) and R03 (motor execution, ~2 min). 64 channels, 160 Hz, no hardware filters, EDF+. About 5 MB of downloads.
  • The official BIDS mirror of the same dataset: OpenNeuro ds004362 (CC0), read through its public S3 bucket — sidecars only, nothing heavy.

The documented defects of ds-eegbci (subjects S088, S089, S092, S100 with inconsistent event timestamps; S038 and S104 also commonly dropped; Sharbrough-style channel labels) are handled explicitly in section 1 and printed in the final cell.

In [1]:
# Setup: dependencies, the shared helpers, non-interactive plotting.
import importlib.util
import subprocess
import sys
import warnings
from pathlib import Path

# 1. Dependencies are pinned in notebooks/requirements.txt.  Nothing is installed
#    when the pinned stack is already present (local runs, CI); a fresh Colab or
#    Binder kernel installs it once.  On Colab, run from a clone of the repository
#    so that notebooks/_shared/ is available (repository URL: TODO(confirm), spec
#    section 13 item 3).
_needed = ("mne", "mne_bids", "scipy", "matplotlib", "pooch")
_missing = [p for p in _needed if importlib.util.find_spec(p) is None]
if _missing:
    _req = next((d / "requirements.txt" for d in (Path.cwd(), *Path.cwd().parents)
                 if (d / "requirements.txt").exists()), None)
    _cmd = [sys.executable, "-m", "pip", "install", "-q"]
    _cmd += ["-r", str(_req)] if _req else ["mne==1.10.2", "pooch>=1.8"]
    if "mne_bids" in _missing:
        _cmd += ["mne-bids==0.19.0"]
    subprocess.check_call(_cmd)

# 2. Shared helpers (notebooks/_shared/helpers.py and helpers_l2.py), located
#    relative to the working directory -- notebooks/<level>/ or notebooks/ --
#    never through an absolute path.
_shared = next((d / "_shared" for d in (Path.cwd(), *Path.cwd().parents)
                if (d / "_shared" / "helpers_l2.py").exists()), None)
if _shared is None:
    raise FileNotFoundError("start the kernel in notebooks/L2/ (or notebooks/) so that _shared/helpers_l2.py is found")
sys.path.insert(0, str(_shared))
import helpers
import helpers_l2 as l2

# 3. Plotting: Jupyter's default inline backend renders static PNGs through Agg
#    (no windows, nothing blocks); outside Jupyter the helpers select Agg.  Every
#    MNE figure is requested with show=False, and plt.show() renders each cell's
#    figures in place.
import matplotlib.pyplot as plt
import numpy as np
import mne
import mne_bids

# Warnings are worth reading, so they are not silenced -- but their default format prints the
# absolute path of the file that raised them, which is nobody else's business and would put this
# machine's directory layout into the saved outputs.  Only the class and the message are shown.
warnings.formatwarning = lambda message, category, *a, **k: f"{category.__name__}: {message}\n"

mne.viz.set_browser_backend("matplotlib", verbose=False)
mne.set_log_level("WARNING")
plt.rcParams["figure.dpi"] = 72
print(f"MNE {mne.__version__}, mne-bids {mne_bids.__version__}; helpers imported from notebooks/_shared; "
      "downloads go to MNE's default data directory unless EEG_COURSE_DATA is set")
MNE 1.10.2, mne-bids 0.19.0; helpers imported from notebooks/_shared; downloads go to MNE's default data directory unless EEG_COURSE_DATA is set

1. Read the file as it is stored

helpers.load_spine standardises this dataset for you (it strips the trailing dots from the Sharbrough labels, attaches standard_1005 and refuses the documented defective subjects). That is the right default for every other notebook and exactly the wrong thing here, because the point of this lesson is what the file actually contains. So the EDF path is resolved through MNE's own downloader and read directly.

In [2]:
from mne.datasets import eegbci

SUBJECT, RUNS = 1, [1, 2, 3]          # S001, R01 eyes open / R02 eyes closed / R03 motor execution
RUN_MEANING = {1: helpers.EEGBCI_RUNS["R01"], 2: helpers.EEGBCI_RUNS["R02"], 3: helpers.EEGBCI_RUNS["R03"]}

with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    edf_paths = eegbci.load_data(SUBJECT, RUNS, update_path=False, verbose=False)
print(f"{len(edf_paths)} EDF files for S{SUBJECT:03d}, runs {RUNS}: "
      + ", ".join(Path(p).name for p in edf_paths))
print(f"excluded by the catalog and refused by helpers.load_spine: {', '.join(helpers.EEGBCI_EXCLUDE_DEFAULT)} "
      f"(inconsistent/overlapping event timestamps); {', '.join(helpers.EEGBCI_EXCLUDE_OPTIONAL)} "
      "(several reports additionally drop them). S001 is in neither list.")

raws_native = []
for p, r in zip(edf_paths, RUNS):
    with warnings.catch_warnings(record=True) as caught:
        warnings.simplefilter("always")
        raw = mne.io.read_raw_edf(p, preload=True, verbose=False)
    raws_native.append(raw)
    print(f"\nR{r:02d} -- {RUN_MEANING[r]}")
    print(f"  sfreq {raw.info['sfreq']:g} Hz | {len(raw.ch_names)} channels | {raw.times[-1] + 1 / raw.info['sfreq']:.1f} s "
          f"| header highpass {raw.info['highpass']:g} Hz, lowpass {raw.info['lowpass']:g} Hz")
    print(f"  channel types: {sorted(set(raw.get_channel_types()))}")
    print(f"  first six names as stored: {raw.ch_names[:6]}")
    print(f"  annotations: {len(raw.annotations)} -> {sorted(set(raw.annotations.description))}")
    for c in caught:
        print("  reader warns:", str(c.message)[:120])
3 EDF files for S001, runs [1, 2, 3]: S001R01.edf, S001R02.edf, S001R03.edf
excluded by the catalog and refused by helpers.load_spine: S088, S089, S092, S100 (inconsistent/overlapping event timestamps); S038, S104 (several reports additionally drop them). S001 is in neither list.
R01 -- Baseline: eyes open (~1 min)
  sfreq 160 Hz | 64 channels | 61.0 s | header highpass 0 Hz, lowpass 80 Hz
  channel types: ['eeg']
  first six names as stored: ['Fc5.', 'Fc3.', 'Fc1.', 'Fcz.', 'Fc2.', 'Fc4.']
  annotations: 1 -> [np.str_('T0')]

R02 -- Baseline: eyes closed (~1 min)
  sfreq 160 Hz | 64 channels | 61.0 s | header highpass 0 Hz, lowpass 80 Hz
  channel types: ['eeg']
  first six names as stored: ['Fc5.', 'Fc3.', 'Fc1.', 'Fcz.', 'Fc2.', 'Fc4.']
  annotations: 1 -> [np.str_('T0')]

R03 -- Motor execution: open/close left or right fist (~2 min)
  sfreq 160 Hz | 64 channels | 125.0 s | header highpass 0 Hz, lowpass 80 Hz
  channel types: ['eeg']
  first six names as stored: ['Fc5.', 'Fc3.', 'Fc1.', 'Fcz.', 'Fc2.', 'Fc4.']
  annotations: 30 -> [np.str_('T0'), np.str_('T1'), np.str_('T2')]

Three readings from that output, and all three are decisions waiting to be made.

The names carry punctuation. Fc5., Fp1., Af7. are Sharbrough-style labels with trailing dots; no standard montage knows them, and set_montage with the default on_missing='warn' will attach nothing and tell you so in a warning you will not read in a loop over 100 subjects. The cell below shows both outcomes.

Every channel is typed eeg. EDF does not record channel types, so MNE assigns the default. This dataset genuinely has 64 EEG channels and nothing else, so the default happens to be right — but "happens to be right" is not the same as "was checked", and on a file with an EOG or a trigger line it would be wrong (L2.1).

The header's filter fields read 0 Hz / Nyquist. That is this dataset telling the truth: it was recorded with no hardware filters, which is why the site uses it for drift and line noise (L1.6, L2.4).

In [3]:
montage = mne.channels.make_standard_montage("standard_1005")
raw = raws_native[2].copy()          # R03: the run with real events

try:
    raw.copy().set_montage(montage, on_missing="raise")
except ValueError as e:
    print("set_montage(on_missing='raise') before renaming:", str(e)[:160], "...")

eegbci.standardize(raw)               # strip the trailing dots, re-case to 10-10 names
print(f"\nafter eegbci.standardize: {raw.ch_names[:6]} ...")
raw.set_montage(montage, on_missing="raise")
pos = raw.get_montage().get_positions()["ch_pos"]
print(f"set_montage(on_missing='raise') passes: {len(pos)} of {len(raw.ch_names)} channels have a position")

raw.set_channel_types({c: "eeg" for c in raw.ch_names})     # the decision, made explicitly
print(f"channel types after the explicit decision: {sorted(set(raw.get_channel_types()))}")

events, event_id = mne.events_from_annotations(raw, verbose=False)
onsets = events[:, 0] / raw.info["sfreq"]
print(f"\nR03 events: {len(events)} in {len(event_id)} classes {event_id}")
for name, code in event_id.items():
    print(f"  {name}: {(events[:, 2] == code).sum():3d} events")
print(f"  first onset {onsets[0]:.2f} s, last {onsets[-1]:.2f} s of {raw.times[-1]:.1f} s; "
      f"inter-event intervals: median {np.median(np.diff(onsets)):.2f} s, "
      f"min {np.diff(onsets).min():.2f} s, max {np.diff(onsets).max():.2f} s")
set_montage(on_missing='raise') before renaming: DigMontage is only a subset of info. There are 64 channel positions not present in the DigMontage. The channels missing from the montage are:

['Fc5.', 'Fc3.',  ...

after eegbci.standardize: ['FC5', 'FC3', 'FC1', 'FCz', 'FC2', 'FC4'] ...
set_montage(on_missing='raise') passes: 64 of 64 channels have a position
channel types after the explicit decision: ['eeg']

R03 events: 30 in 3 classes {np.str_('T0'): 1, np.str_('T1'): 2, np.str_('T2'): 3}
  T0:  15 events
  T1:   8 events
  T2:   7 events
  first onset 0.00 s, last 120.40 s of 125.0 s; inter-event intervals: median 4.20 s, min 4.10 s, max 4.20 s
In [4]:
fig, axes = plt.subplots(1, 2, figsize=(13, 4.2), gridspec_kw=dict(width_ratios=[1, 1.6]))
raw.plot_sensors(show_names=["Fp1", "Fz", "Cz", "Pz", "Oz", "C3", "C4"], axes=axes[0], show=False)
axes[0].set_title("S001 after renaming: standard_1005 positions (64 of 64 matched)")
helpers.plot_traces(raw, ["Fp1", "F3", "C3", "Cz", "C4", "P3", "Pz", "O1"], t0=10, duration=10,
                    spacing_uV=100, ax=axes[1], title="S001 R03, 10 s of raw EDF (uV)")
fig.tight_layout()
plt.show()   # render the static figure(s) of this cell inline
Figure 1 of notebook nb-2-1-bids, an output plot. The text around it states what it shows and the units of every axis.

2. The messy folder

The exercise asks you to BIDS-ify a messy folder. This is what one looks like: the same three runs, saved under names that mean something only to the person who made them, with the metadata in a free-text file instead of a sidecar. Nothing here is a strawman — file names like these are what most lab archives actually contain.

The folder is built in a temporary directory so that the notebook leaves nothing behind; a learner working through the exercise can point the same code at their own folder.

In [5]:
import shutil
import tempfile

_tmp = tempfile.TemporaryDirectory(prefix="nb-2-1-")
WORK = Path(_tmp.name)
MESSY, BIDS_ROOT = WORK / "messy_folder", WORK / "bids_out"
MESSY.mkdir(parents=True)

MESSY_NAMES = {1: "subj1_eyesopen_FINAL.edf", 2: "subj1_eyesclosed_FINAL.edf", 3: "subj1_task_run3(copy).edf"}
for p, r in zip(edf_paths, RUNS):
    shutil.copyfile(p, MESSY / MESSY_NAMES[r])
(MESSY / "notes about the recordings.txt").write_text(
    "S001, 64ch cap, 160Hz. amp had no filters on. mains 60Hz here.\n"
    "ref = ear lobe (left or right, ask the RA)\n"
    "run3 = open/close fist task, T0 rest T1 left T2 right\n"
    "units are microvolts I think\n", encoding="utf-8")

print("the messy folder:")
for f in sorted(MESSY.iterdir()):
    print(f"  {f.name:34s} {f.stat().st_size / 1e6:6.2f} MB")
print("\nwhat a BIDS writer will ask for that this folder does not answer in machine-readable form:")
for q in ("task name", "SamplingFrequency", "PowerLineFrequency", "EEGReference", "SoftwareFilters",
          "channel types (EEG/EOG/ECG/stim/misc)", "channel units", "event onsets in seconds + condition names",
          "participant id and any demographics", "dataset name, licence, authors"):
    print(f"  - {q}")
the messy folder:
  notes about the recordings.txt       0.00 MB
  subj1_eyesclosed_FINAL.edf           1.28 MB
  subj1_eyesopen_FINAL.edf             1.28 MB
  subj1_task_run3(copy).edf            2.60 MB

what a BIDS writer will ask for that this folder does not answer in machine-readable form:
  - task name
  - SamplingFrequency
  - PowerLineFrequency
  - EEGReference
  - SoftwareFilters
  - channel types (EEG/EOG/ECG/stim/misc)
  - channel units
  - event onsets in seconds + condition names
  - participant id and any demographics
  - dataset name, licence, authors

3. Convert with mne-bids

Every argument below is a fact that had to be established first, which is the whole point of the exercise (the lesson's note: converting to BIDS is a forced metadata audit). Where the fact comes from the site's dataset catalog rather than from the file, it is marked in the printed table.

Two mechanical points. write_raw_bids converts formats according to what BIDS allows for EEG; here the preloaded EDF is written as BrainVision, so the file that comes out is not the file that went in, and the conversion is recorded. And the events are already in the Raw as annotations, so they are not passed again through events= — passing both writes them twice and the round trip comes back with double the annotations.

In [6]:
from mne_bids import BIDSPath, write_raw_bids, read_raw_bids, print_dir_tree

TASK = {1: "eyesopen", 2: "eyesclosed", 3: "motorexec"}
LINE_FREQ = helpers.DATASETS["ds-eegbci"]["mains_hz"]                 # 60 Hz, from the catalog
REFERENCE = "left or right ear lobe (catalog: TODO(confirm) which)"   # catalog + the mirror's own sidecar
POWERLINE_SOURCE = "catalog (data/catalog/datasets/eegmmidb.md), not the file"

written = {}
for messy_file, r in [(MESSY / MESSY_NAMES[r], r) for r in RUNS]:
    raw_m = mne.io.read_raw_edf(messy_file, preload=True, verbose=False)
    eegbci.standardize(raw_m)                                     # names first (L2.1)
    raw_m.set_montage(montage, on_missing="raise")                # positions second
    raw_m.set_channel_types({c: "eeg" for c in raw_m.ch_names})   # types are a decision
    raw_m.info["line_freq"] = LINE_FREQ                           # required by BIDS; not in the file
    bp = BIDSPath(subject=f"{SUBJECT:03d}", task=TASK[r], run=f"{r:02d}", datatype="eeg", root=BIDS_ROOT)
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        written[r] = write_raw_bids(raw_m, bp, format="BrainVision", allow_preload=True,
                                    overwrite=True, verbose=False)
    # mne-bids cannot invent the reference or the software filters; they are written afterwards.
    mne_bids.update_sidecar_json(bids_path=written[r].copy().update(extension=".json"),
                                 entries={"EEGReference": REFERENCE, "SoftwareFilters": "n/a",
                                          "HardwareFilters": "n/a (the catalog documents no online filtering "
                                                             "and no notch)",
                                          "EEGPlacementScheme": "10-10 (extended 10-20)",
                                          "Manufacturer": "n/a (BCI2000 acquisition; TODO(confirm) the amplifier)",
                                          "InstitutionName": "n/a"},
                                 verbose=False)
    print(f"R{r:02d} -> {written[r].basename}  ({len(raw_m.ch_names)} ch, {len(raw_m.annotations)} annotations)")

# The two sidecars BIDS asks for that mne-bids cannot fill from the data.
import json
(BIDS_ROOT / "dataset_description.json").write_text(json.dumps({
    "Name": "EEG Motor Movement/Imagery Dataset -- three-run subset converted for nb-2-1",
    "BIDSVersion": "1.8.0",
    "License": helpers.DATASETS["ds-eegbci"]["license"],
    "Authors": ["Schalk, G.", "McFarland, D. J.", "Hinterberger, T.", "Birbaumer, N.", "Wolpaw, J. R."],
    "ReferencesAndLinks": [helpers.DATASETS["ds-eegbci"]["source"]],
    "DatasetDOI": f"doi:{helpers.DATASETS['ds-eegbci']['doi']}",
}, indent=2), encoding="utf-8")
(BIDS_ROOT / "participants.json").write_text(json.dumps({
    "participant_id": {"Description": "Unique participant identifier"},
    "age": {"Description": "Age in years", "Units": "years"},
    "sex": {"Description": "Biological sex", "Levels": {"M": "male", "F": "female", "n/a": "not documented"}},
    "handedness": {"Description": "Self-reported handedness"},
}, indent=2), encoding="utf-8")
print("\nwrote dataset_description.json and participants.json by hand: a writer cannot invent a licence.")
for lock in BIDS_ROOT.rglob("*.lock"):      # mne-bids leaves file locks behind; they are not BIDS files
    lock.unlink()
print_dir_tree(BIDS_ROOT, max_depth=3)
R01 -> sub-001_task-eyesopen_run-01_eeg.vhdr  (64 ch, 1 annotations)
R02 -> sub-001_task-eyesclosed_run-02_eeg.vhdr  (64 ch, 1 annotations)
R03 -> sub-001_task-motorexec_run-03_eeg.vhdr  (64 ch, 30 annotations)

wrote dataset_description.json and participants.json by hand: a writer cannot invent a licence.
|bids_out/
|--- README
|--- dataset_description.json
|--- participants.json
|--- participants.tsv
|--- sub-001/
|------ sub-001_scans.tsv
|------ eeg/
|--------- sub-001_space-CapTrak_coordsystem.json
|--------- sub-001_space-CapTrak_electrodes.json
|--------- sub-001_space-CapTrak_electrodes.tsv
|--------- sub-001_task-eyesclosed_run-02_channels.tsv
|--------- sub-001_task-eyesclosed_run-02_eeg.eeg
|--------- sub-001_task-eyesclosed_run-02_eeg.json
|--------- sub-001_task-eyesclosed_run-02_eeg.vhdr
|--------- sub-001_task-eyesclosed_run-02_eeg.vmrk
|--------- sub-001_task-eyesclosed_run-02_events.json
|--------- sub-001_task-eyesclosed_run-02_events.tsv
|--------- sub-001_task-eyesopen_run-01_channels.tsv
|--------- sub-001_task-eyesopen_run-01_eeg.eeg
|--------- sub-001_task-eyesopen_run-01_eeg.json
|--------- sub-001_task-eyesopen_run-01_eeg.vhdr
|--------- sub-001_task-eyesopen_run-01_eeg.vmrk
|--------- sub-001_task-eyesopen_run-01_events.json
|--------- sub-001_task-eyesopen_run-01_events.tsv
|--------- sub-001_task-motorexec_run-03_channels.tsv
|--------- sub-001_task-motorexec_run-03_eeg.eeg
|--------- sub-001_task-motorexec_run-03_eeg.json
|--------- sub-001_task-motorexec_run-03_eeg.vhdr
|--------- sub-001_task-motorexec_run-03_eeg.vmrk
|--------- sub-001_task-motorexec_run-03_events.json
|--------- sub-001_task-motorexec_run-03_events.tsv
In [7]:
sidecar = json.loads((written[3].copy().update(extension=".json")).fpath.read_text())
rows = [[k, str(v), ("the file" if k in ("SamplingFrequency", "RecordingDuration", "EEGChannelCount",
                                         "RecordingType", "EOGChannelCount", "ECGChannelCount",
                                         "EMGChannelCount", "MiscChannelCount") else "you")]
        for k, v in sidecar.items()]
print("the _eeg.json for R03, and where each value came from:\n")
print(l2.fmt_table([{"field": k, "value": v[:60], "source": s} for k, v, s in rows],
                   ["field", "value", "source"]))
print(f"\nPowerLineFrequency came from the {POWERLINE_SOURCE}.")
print(f"EEGReference as recorded here: {REFERENCE!r} -- mne-bids writes 'n/a' unless you set it, "
      "and 'n/a' is an honest answer only if you genuinely do not know.")
channels_tsv = (written[3].copy().update(suffix="channels", extension=".tsv")).fpath.read_text().splitlines()
print(f"\n_channels.tsv header: {channels_tsv[0]}")
print(f"first three rows:      " + " | ".join(channels_tsv[1:4]))
events_tsv = (written[3].copy().update(suffix="events", extension=".tsv")).fpath.read_text().splitlines()
print(f"\n_events.tsv: {len(events_tsv) - 1} rows, header {events_tsv[0]}")
print("first three rows:      " + " | ".join(events_tsv[1:4]))
the _eeg.json for R03, and where each value came from:

field                value                                                         source  
-------------------  ------------------------------------------------------------  --------
TaskName             motorexec                                                     you     
Manufacturer         n/a (BCI2000 acquisition; TODO(confirm) the amplifier)        you     
PowerLineFrequency   60.0                                                          you     
SamplingFrequency    160.0                                                         the file
SoftwareFilters      n/a                                                           you     
RecordingDuration    124.99375                                                     the file
RecordingType        continuous                                                    the file
EEGReference         left or right ear lobe (catalog: TODO(confirm) which)         you     
EEGGround            n/a                                                           you     
EEGPlacementScheme   10-10 (extended 10-20)                                        you     
EEGChannelCount      64                                                            the file
EOGChannelCount      0                                                             the file
ECGChannelCount      0                                                             the file
EMGChannelCount      0                                                             the file
MiscChannelCount     0                                                             the file
TriggerChannelCount  0                                                             you     
HardwareFilters      n/a (the catalog documents no online filtering and no notch)  you     
InstitutionName      n/a                                                           you     

PowerLineFrequency came from the catalog (data/catalog/datasets/eegmmidb.md), not the file.
EEGReference as recorded here: 'left or right ear lobe (catalog: TODO(confirm) which)' -- mne-bids writes 'n/a' unless you set it, and 'n/a' is an honest answer only if you genuinely do not know.

_channels.tsv header: name	type	units	low_cutoff	high_cutoff	description	sampling_frequency	status	status_description
first three rows:      FC5	EEG	µV	0.0	80.0	ElectroEncephaloGram	160.0	good	n/a | FC3	EEG	µV	0.0	80.0	ElectroEncephaloGram	160.0	good	n/a | FC1	EEG	µV	0.0	80.0	ElectroEncephaloGram	160.0	good	n/a

_events.tsv: 30 rows, header onset	duration	trial_type	value	sample
first three rows:      0.0	4.2	T0	1	0 | 4.2	4.1	T2	3	672 | 8.3	4.2	T0	1	1328

4. The round trip

read_raw_bids should hand back the same recording: the same channel names in the same order, the same types, the same sampling rate and the same events. This is the test that matters, and it is worth running on every conversion — a silent change here becomes an unexplainable group result later.

In [8]:
ROUND_TRIP = {}
for r in RUNS:
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        back = read_raw_bids(written[r], verbose=False)
    orig = raws_native[RUNS.index(r)].copy()
    eegbci.standardize(orig)
    ev_back, id_back = mne.events_from_annotations(back, verbose=False)
    ev_orig, id_orig = mne.events_from_annotations(orig, verbose=False)
    checks = {
        "channel names": back.ch_names == orig.ch_names,
        "channel types": back.get_channel_types() == orig.get_channel_types(),
        "sampling rate": back.info["sfreq"] == orig.info["sfreq"],
        "n samples": back.n_times == orig.n_times,
        "n events": len(ev_back) == len(ev_orig),
        "event classes": set(id_back) == set(id_orig),
        "data (max abs difference < 1 nV)": float(np.abs(back.get_data() - orig.get_data()).max()) < 1e-9,
    }
    ROUND_TRIP[r] = dict(checks, n_events=len(ev_back),
                         max_abs_diff_uV=float(np.abs(back.get_data() - orig.get_data()).max()) * 1e6)
    status = "OK" if all(checks.values()) else "MISMATCH"
    print(f"R{r:02d} round trip: {status:8s} {len(ev_back):2d} events, "
          f"max |difference| {ROUND_TRIP[r]['max_abs_diff_uV']:.4g} uV, "
          + ", ".join(k for k, v in checks.items() if not v))
R01 round trip: OK        1 events, max |difference| 0 uV, 
R02 round trip: OK        1 events, max |difference| 0 uV, 
R03 round trip: OK       30 events, max |difference| 0 uV, 

The data survive the EDF → BrainVision conversion to well below a nanovolt, which is the numerical precision of the format change and not a fact about EEG. What does not survive automatically is anything the source format never held: the reference, the line frequency, the channel types. Those are in the sidecar because you put them there.

5. Compare with the official mirror

The same dataset was converted to BIDS by somebody else and published as OpenNeuro ds004362 (CC0). That makes this lesson checkable: two independent conversions of one recording, and every difference is either a choice or a mistake. Only sidecars are downloaded — a few kilobytes — through OpenNeuro's public S3 bucket.

In [9]:
import urllib.request

MIRROR = "ds004362"
MIRROR_BASE = f"https://s3.amazonaws.com/openneuro.org/{MIRROR}/"
# The mirror files every run under one task name with an unpadded run entity.
MIRROR_FILES = {r: f"sub-001/eeg/sub-001_task-motion_run-{r}_" for r in RUNS}


def fetch_text(name, timeout=30):
    """Fetch one small text file from the mirror; returns None (with a note) if it cannot be read."""
    try:
        with urllib.request.urlopen(MIRROR_BASE + name, timeout=timeout) as resp:
            return resp.read().decode("utf-8")
    except Exception as e:
        print(f"  could not read {name}: {type(e).__name__}: {e}")
        return None


mirror = {}
for r in RUNS:
    mirror[r] = {kind: fetch_text(MIRROR_FILES[r] + suffix)
                 for kind, suffix in (("eeg.json", "eeg.json"), ("events.tsv", "events.tsv"),
                                      ("channels.tsv", "channels.tsv"))}
mirror_dd = fetch_text("dataset_description.json")
if mirror_dd:
    dd = json.loads(mirror_dd)
    print(f"{MIRROR}: {dd['Name']!r}, BIDS {dd['BIDSVersion']}, licence {dd['License']}, "
          f"DOI {dd.get('DatasetDOI', 'n/a')}")
MIRROR_EVENT_COUNTS, MIRROR_LABELS = {}, {}
for r in RUNS:
    lines = [l for l in (mirror[r]["events.tsv"] or "").splitlines()[1:] if l.strip()]
    MIRROR_EVENT_COUNTS[r] = len(lines) if mirror[r]["events.tsv"] else None
    MIRROR_LABELS[r] = sorted({l.split("	")[-1] for l in lines})
    print(f"  R{r:02d}: mirror _events.tsv has {MIRROR_EVENT_COUNTS[r]} rows labelled {MIRROR_LABELS[r]}; "
          f"our conversion has {ROUND_TRIP[r]['n_events']} rows labelled "
          f"{sorted(set(raws_native[RUNS.index(r)].annotations.description))}")
ds004362: 'EEG Motor Movement/Imagery Dataset', BIDS 1.8.0, licence CC0, DOI doi:10.18112/openneuro.ds004362.v1.0.0
  R01: mirror _events.tsv has 1 rows labelled ['BASE1T0']; our conversion has 1 rows labelled [np.str_('T0')]
  R02: mirror _events.tsv has 1 rows labelled ['BASE2T0']; our conversion has 1 rows labelled [np.str_('T0')]
  R03: mirror _events.tsv has 30 rows labelled ['TASK1T0', 'TASK1T1', 'TASK1T2']; our conversion has 30 rows labelled [np.str_('T0'), np.str_('T1'), np.str_('T2')]
In [10]:
r = 3                                        # the run with real events
mine_json = json.loads((written[r].copy().update(extension=".json")).fpath.read_text())
theirs_json = json.loads(mirror[r]["eeg.json"]) if mirror[r]["eeg.json"] else {}
keys = sorted(set(mine_json) | set(theirs_json))
rows = [{"field": k, "ours": str(mine_json.get(k, "-"))[:44], "mirror": str(theirs_json.get(k, "-"))[:44],
         "same": "yes" if mine_json.get(k) == theirs_json.get(k) else "NO"} for k in keys]
print(f"_eeg.json for run {r}, ours versus {MIRROR}:\n")
print(l2.fmt_table(rows, ["field", "ours", "mirror", "same"]))

mine_ch = [l.split("\t") for l in (written[r].copy().update(suffix="channels", extension=".tsv")
                                   ).fpath.read_text().splitlines()]
their_ch = [l.split("\t") for l in mirror[r]["channels.tsv"].splitlines()] if mirror[r]["channels.tsv"] else []
mine_names = [row[0] for row in mine_ch[1:]]
their_names = [row[0] for row in their_ch[1:]]
print(f"\n_channels.tsv: ours {len(mine_names)} rows {mine_ch[0]}, mirror {len(their_names)} rows {their_ch[0]}")
print(f"  same order, case-insensitively: {[n.lower() for n in mine_names] == [n.lower() for n in their_names]}")
print(f"  names that differ in spelling: "
      f"{[(a, b) for a, b in zip(mine_names, their_names) if a != b][:5]} ... "
      f"({sum(a != b for a, b in zip(mine_names, their_names))} of {len(mine_names)})")
their_types = {row[1] for row in their_ch[1:]}
their_units = {row[2] for row in their_ch[1:]}
mine_types = {row[mine_ch[0].index("type")] for row in mine_ch[1:]}
mine_units = {row[mine_ch[0].index("units")] for row in mine_ch[1:]}
print(f"  types  -- ours {sorted(mine_types)}, mirror {sorted(their_types)}")
print(f"  units  -- ours {sorted(mine_units)}, mirror {sorted(their_units)}")
_eeg.json for run 3, ours versus ds004362:

field                ours                                          mirror                                        same
-------------------  --------------------------------------------  --------------------------------------------  ----
CapManufacturer      -                                             Electro-Cap International, Inc.               NO  
ECGChannelCount      0                                             0                                             yes 
EEGChannelCount      64                                            64                                            yes 
EEGGround            n/a                                           -                                             NO  
EEGPlacementScheme   10-10 (extended 10-20)                        10-10                                         NO  
EEGReference         left or right ear lobe (catalog: TODO(confir  Left or right ear lobe                        NO  
EMGChannelCount      0                                             0                                             yes 
EOGChannelCount      0                                             0                                             yes 
HardwareFilters      n/a (the catalog documents no online filteri  -                                             NO  
InstitutionName      n/a                                           -                                             NO  
Manufacturer         n/a (BCI2000 acquisition; TODO(confirm) the   -                                             NO  
MiscChannelCount     0                                             -                                             NO  
PowerLineFrequency   60.0                                          60                                            yes 
RecordingDuration    124.99375                                     125                                           NO  
RecordingType        continuous                                    continuous                                    yes 
SamplingFrequency    160.0                                         160                                           yes 
SoftwareFilters      n/a                                           {'FilterDescription': {'Description': 'N/A'}  NO  
TaskName             motorexec                                     motion                                        NO  
TriggerChannelCount  0                                             -                                             NO  

_channels.tsv: ours 64 rows ['name', 'type', 'units', 'low_cutoff', 'high_cutoff', 'description', 'sampling_frequency', 'status', 'status_description'], mirror 64 rows ['name', 'type', 'units']
  same order, case-insensitively: True
  names that differ in spelling: [('FC5', 'Fc5'), ('FC3', 'Fc3'), ('FC1', 'Fc1'), ('FCz', 'Fcz'), ('FC2', 'Fc2')] ... (28 of 64)
  types  -- ours ['EEG'], mirror ['n/a']
  units  -- ours ['µV'], mirror ['n/a']

This is the finding the exercise is built on, and it is not that one conversion is wrong.

  • The event counts agree, which is the substantive check: both conversions found the same number of events in the same run, so neither lost a trial.
  • The mirror's _channels.tsv leaves type and units as n/a. BIDS allows it, and the validator will not complain, but a pipeline that reads units to decide whether the numbers are volts or microvolts gets nothing — the factor-of-a-million error the lesson warns about.
  • The event labels differ: the mirror writes BASE1T0, TASK1T0, TASK1T1 — the run context folded into the label — where a conversion from the annotations writes the bare T0, T1, T2 that the EDF stores. Both are faithful; only one of them tells you which run a row came from if the files are ever concatenated.
  • The mirror's EEGReference reads "Left or right ear lobe", which is exactly the free text the lesson describes: it records genuine uncertainty rather than hiding it.
  • Our sidecar reports EEGChannelCount and a RecordingDuration read from the data; the mirror's values come from its own converter. Any disagreement here is worth chasing, because it means the two files are not the same recording.

6. The validator's job, and what can be checked here

The official BIDS validator is a separate tool and is not installed by this notebook. What can be checked in Python is the structural part of its job — the files that must exist, the entities that must match, the columns that must be present — and that is what the cell below does. Run the real validator on your own conversion as well; it catches naming mistakes this check does not.

In [11]:
def structural_checks(root: Path) -> list[dict]:
    """A subset of what the BIDS validator checks. Not a substitute for it."""
    out = []

    def add(item, ok, value="", note=""):
        out.append({"check": item, "status": "ok" if ok else "FAIL", "value": str(value)[:52], "note": note})

    add("dataset_description.json exists", (root / "dataset_description.json").exists())
    if (root / "dataset_description.json").exists():
        dd = json.loads((root / "dataset_description.json").read_text())
        for k in ("Name", "BIDSVersion", "License"):
            add(f"dataset_description.{k}", k in dd, dd.get(k, ""))
    add("participants.tsv exists", (root / "participants.tsv").exists())
    add("participants.json describes the columns", (root / "participants.json").exists())
    add("README exists", (root / "README").exists())
    subs = sorted(d.name for d in root.iterdir() if d.is_dir() and d.name.startswith("sub-"))
    add("at least one sub-<label> directory", bool(subs), ", ".join(subs))
    if (root / "participants.tsv").exists():
        ids = [l.split("\t")[0] for l in (root / "participants.tsv").read_text().splitlines()[1:] if l.strip()]
        add("every subject directory has a participants.tsv row", set(subs) <= set(ids), ", ".join(ids))
    for eeg_json in sorted(root.rglob("*_eeg.json")):
        stem = eeg_json.name[: -len("_eeg.json")]
        j = json.loads(eeg_json.read_text())
        missing = [k for k in ("TaskName", "SamplingFrequency", "PowerLineFrequency", "EEGReference",
                               "SoftwareFilters") if k not in j or j[k] in (None, "")]
        vague = [k for k in ("EEGReference",) if str(j.get(k)).strip().lower() == "n/a"]
        add(f"{stem}: required _eeg.json fields", not missing,
            "missing: " + ", ".join(missing) if missing else "all present")
        add(f"{stem}: EEGReference is not a bare 'n/a'", not vague,
            "n/a" if vague else str(j.get("EEGReference"))[:44],
            "BIDS allows n/a; it is honest only when you genuinely do not know. "
            "SoftwareFilters 'n/a' here IS the fact: no software filter was applied, and HardwareFilters "
            "records that the amplifier applied none either")
        for suffix, cols in (("channels.tsv", ("name", "type", "units")), ("events.tsv", ("onset", "duration"))):
            f = eeg_json.with_name(f"{stem}_{suffix}")
            header = f.read_text().splitlines()[0].split("\t") if f.exists() else []
            add(f"{stem}: {suffix} columns", f.exists() and all(c in header for c in cols), ", ".join(header))
        add(f"{stem}: PowerLineFrequency is a number, not n/a",
            isinstance(j.get("PowerLineFrequency"), (int, float)), j.get("PowerLineFrequency"))
    return out


checks = structural_checks(BIDS_ROOT)
print(l2.fmt_table(checks, ["check", "status", "value"]))
n_fail = sum(c["status"] == "FAIL" for c in checks)
print(f"\n{len(checks) - n_fail} of {len(checks)} structural checks pass; {n_fail} fail.")
print("This is not the BIDS validator. Run it on your own output and read every warning: "
      "https://bids-standard.github.io/bids-validator/")
check                                                                    status  value                                               
-----------------------------------------------------------------------  ------  ----------------------------------------------------
dataset_description.json exists                                          ok                                                          
dataset_description.Name                                                 ok      EEG Motor Movement/Imagery Dataset -- three-run subs
dataset_description.BIDSVersion                                          ok      1.8.0                                               
dataset_description.License                                              ok      ODC-By 1.0                                          
participants.tsv exists                                                  ok                                                          
participants.json describes the columns                                  ok                                                          
README exists                                                            ok                                                          
at least one sub-<label> directory                                       ok      sub-001                                             
every subject directory has a participants.tsv row                       ok      sub-001                                             
sub-001_task-eyesclosed_run-02: required _eeg.json fields                ok      all present                                         
sub-001_task-eyesclosed_run-02: EEGReference is not a bare 'n/a'         ok      left or right ear lobe (catalog: TODO(confir        
sub-001_task-eyesclosed_run-02: channels.tsv columns                     ok      name, type, units, low_cutoff, high_cutoff, descript
sub-001_task-eyesclosed_run-02: events.tsv columns                       ok      onset, duration, trial_type, value, sample          
sub-001_task-eyesclosed_run-02: PowerLineFrequency is a number, not n/a  ok      60.0                                                
sub-001_task-eyesopen_run-01: required _eeg.json fields                  ok      all present                                         
sub-001_task-eyesopen_run-01: EEGReference is not a bare 'n/a'           ok      left or right ear lobe (catalog: TODO(confir        
sub-001_task-eyesopen_run-01: channels.tsv columns                       ok      name, type, units, low_cutoff, high_cutoff, descript
sub-001_task-eyesopen_run-01: events.tsv columns                         ok      onset, duration, trial_type, value, sample          
sub-001_task-eyesopen_run-01: PowerLineFrequency is a number, not n/a    ok      60.0                                                
sub-001_task-motorexec_run-03: required _eeg.json fields                 ok      all present                                         
sub-001_task-motorexec_run-03: EEGReference is not a bare 'n/a'          ok      left or right ear lobe (catalog: TODO(confir        
sub-001_task-motorexec_run-03: channels.tsv columns                      ok      name, type, units, low_cutoff, high_cutoff, descript
sub-001_task-motorexec_run-03: events.tsv columns                        ok      onset, duration, trial_type, value, sample          
sub-001_task-motorexec_run-03: PowerLineFrequency is a number, not n/a   ok      60.0                                                

24 of 24 structural checks pass; 0 fail.
This is not the BIDS validator. Run it on your own output and read every warning: https://bids-standard.github.io/bids-validator/

7. The numbers

In [12]:
print("nb-2-1-bids -- L2.1 exercise numbers (draft; TODO(confirm) at author review)")
print(f"Data: ds-eegbci S{SUBJECT:03d} runs R01/R02/R03 (PhysioNet DOI {helpers.DATASETS['ds-eegbci']['doi']}; "
      f"{helpers.DATASETS['ds-eegbci']['license']}), converted here with mne-bids {mne_bids.__version__}; "
      f"official BIDS mirror OpenNeuro {MIRROR} (CC0) read for comparison.")
print()
print("ex-2-1-mirror-diff -- how many events does the mirror's _events.tsv contain for the converted run?")
for r in RUNS:
    print(f"  run-{r:02d} ({RUN_MEANING[r].split(':')[0]}): mirror {MIRROR_EVENT_COUNTS[r]} events, "
          f"our conversion {ROUND_TRIP[r]['n_events']} events  "
          f"-> {'agree' if MIRROR_EVENT_COUNTS[r] == ROUND_TRIP[r]['n_events'] else 'DISAGREE'}")
print(f"  The exercise's run is R03 (the one with real events): answer = {MIRROR_EVENT_COUNTS[3]} events "
      "(tolerance 0; a count, and a mismatch with your own conversion is the finding).")
print()
print("Round trip through mne-bids (read_raw_bids against the source EDF):")
for r in RUNS:
    rt = ROUND_TRIP[r]
    print(f"  R{r:02d}: names/types/sfreq/n_samples/events all "
          f"{'unchanged' if all(v for k, v in rt.items() if isinstance(v, bool)) else 'CHANGED'}; "
          f"max |difference| {rt['max_abs_diff_uV']:.3g} uV (EDF -> BrainVision conversion precision)")
print()
print("ex-2-1-bidsify-checklist -- the state of the conversion made here, item by item:")
CHECKLIST = [
    ("Channel names re-cased and de-punctuated; set_montage(on_missing='raise') passes",
     f"yes -- {len(pos)}/{len(raw.ch_names)} positions attached"),
    ("Channel types set explicitly; no trigger line typed as EEG",
     f"yes -- {sorted(set(raw.get_channel_types()))}; this dataset has no EOG/ECG/stim channel"),
    ("dataset_description.json with Name, BIDSVersion, License", "yes -- written by hand"),
    ("participants.tsv one row per subject; participants.json describes every column",
     "yes -- mne-bids writes the tsv, the json was written by hand"),
    ("_eeg.json has SamplingFrequency, PowerLineFrequency, EEGReference, SoftwareFilters",
     f"SamplingFrequency from the file; PowerLineFrequency {LINE_FREQ:g} Hz from the catalog; EEGReference "
     f"{REFERENCE!r} written with update_sidecar_json; SoftwareFilters 'n/a' (the amplifier's own band-pass "
     "is all that happened) -- TODO(confirm) which ear lobe"),
    ("_channels.tsv lists name, type and units, matching what the data holds",
     f"ours {sorted(mine_units)}; the mirror leaves both n/a"),
    ("_events.tsv onsets in seconds; counts per condition match the protocol",
     f"yes -- {len(events_tsv) - 1} rows for R03, same count as the mirror"),
    ("Subjects with documented defects excluded with the reason recorded, or the defect noted",
     f"S088/S089/S092/S100 (inconsistent event timestamps) and S038/S104 refused by helpers.load_spine; "
     f"S{SUBJECT:03d} is in neither list"),
    ("read_raw_bids round-trips names, types, sampling rate and event counts",
     "yes -- all three runs"),
    ("The BIDS validator runs with no errors",
     f"NOT CHECKED here -- {len(checks) - n_fail}/{len(checks)} structural checks pass; "
     "run the official validator yourself"),
]
for item, state in CHECKLIST:
    print(f"  [{'x' if not state.startswith('NOT') else ' '}] {item}\n      {state}")
print()
print("Differences between the two conversions that are choices, not errors: event labels "
      "(mirror 'TASK1T1' vs annotation 'T1'), channel-name case (mirror 'Fc5' vs MNE's 'FC5'), and the "
      "mirror's n/a channel types and units.")
_tmp.cleanup()
print("\ntemporary BIDS folders removed; the EDF downloads stay in MNE's data directory.")
nb-2-1-bids -- L2.1 exercise numbers (draft; TODO(confirm) at author review)
Data: ds-eegbci S001 runs R01/R02/R03 (PhysioNet DOI 10.13026/C28G6P; ODC-By 1.0), converted here with mne-bids 0.19.0; official BIDS mirror OpenNeuro ds004362 (CC0) read for comparison.

ex-2-1-mirror-diff -- how many events does the mirror's _events.tsv contain for the converted run?
  run-01 (Baseline): mirror 1 events, our conversion 1 events  -> agree
  run-02 (Baseline): mirror 1 events, our conversion 1 events  -> agree
  run-03 (Motor execution): mirror 30 events, our conversion 30 events  -> agree
  The exercise's run is R03 (the one with real events): answer = 30 events (tolerance 0; a count, and a mismatch with your own conversion is the finding).

Round trip through mne-bids (read_raw_bids against the source EDF):
  R01: names/types/sfreq/n_samples/events all unchanged; max |difference| 0 uV (EDF -> BrainVision conversion precision)
  R02: names/types/sfreq/n_samples/events all unchanged; max |difference| 0 uV (EDF -> BrainVision conversion precision)
  R03: names/types/sfreq/n_samples/events all unchanged; max |difference| 0 uV (EDF -> BrainVision conversion precision)

ex-2-1-bidsify-checklist -- the state of the conversion made here, item by item:
  [x] Channel names re-cased and de-punctuated; set_montage(on_missing='raise') passes
      yes -- 64/64 positions attached
  [x] Channel types set explicitly; no trigger line typed as EEG
      yes -- ['eeg']; this dataset has no EOG/ECG/stim channel
  [x] dataset_description.json with Name, BIDSVersion, License
      yes -- written by hand
  [x] participants.tsv one row per subject; participants.json describes every column
      yes -- mne-bids writes the tsv, the json was written by hand
  [x] _eeg.json has SamplingFrequency, PowerLineFrequency, EEGReference, SoftwareFilters
      SamplingFrequency from the file; PowerLineFrequency 60 Hz from the catalog; EEGReference 'left or right ear lobe (catalog: TODO(confirm) which)' written with update_sidecar_json; SoftwareFilters 'n/a' (the amplifier's own band-pass is all that happened) -- TODO(confirm) which ear lobe
  [x] _channels.tsv lists name, type and units, matching what the data holds
      ours ['µV']; the mirror leaves both n/a
  [x] _events.tsv onsets in seconds; counts per condition match the protocol
      yes -- 30 rows for R03, same count as the mirror
  [x] Subjects with documented defects excluded with the reason recorded, or the defect noted
      S088/S089/S092/S100 (inconsistent event timestamps) and S038/S104 refused by helpers.load_spine; S001 is in neither list
  [x] read_raw_bids round-trips names, types, sampling rate and event counts
      yes -- all three runs
  [ ] The BIDS validator runs with no errors
      NOT CHECKED here -- 24/24 structural checks pass; run the official validator yourself

Differences between the two conversions that are choices, not errors: event labels (mirror 'TASK1T1' vs annotation 'T1'), channel-name case (mirror 'Fc5' vs MNE's 'FC5'), and the mirror's n/a channel types and units.

temporary BIDS folders removed; the EDF downloads stay in MNE's data directory.