Capstone C4, mu/beta ERD: left against right hand imagery over a documented subset, contralateral lateralisation, a cluster test on the difference, and which subjects have no mu peak, with a FULL_COHORT switch

nb-c4-mu-beta-erd Level 4 · Time-Frequency and Oscillations capstone ~6 min Used in C4 · Capstone — Mu/beta ERD

Downloads from ds-eegbci when you run it.

Download the notebook (.ipynb) Outputs below are the ones stored when it was executed — you do not need to run anything to read it.

nb-c4-mu-beta-erd · Capstone C4, mu/beta ERD

Capstone C4 · Level 4 · Status draft — for expert review; uncertain points carry TODO(confirm).

Brief (spec §6 C4). On ds-eegbci motor-imagery runs R04, R08 and R12, with documented defective subjects excluded and the reason recorded: compute ERD maps for left- versus right-hand imagery for every subject in a documented subset, show contralateral lateralisation at C3 and C4, run a cluster test on the difference, and use spectral parameterisation to verify that a mu peak exists per subject — reporting who lacks one.

Rubric, and where each item is answered.

Rubric item Section
Baseline choice justified and edge-safe 1 and 2
Normalisation stated 2, with the averaging order named
Cluster interpretation correct 5
Subjects without a mu peak handled explicitly 4 and 6

Cohort. FULL_COHORT = False runs the documented subset of ten subjects and is what is executed here. FULL_COHORT = True runs every subject helpers.load_spine will load, which is about 103 of the 109 — roughly 740 MB of download and well outside the §11 ten-minute limit. It is a local switch, not a CI one.

Data. ds-eegbci — EEGMMIDB, Schalk, McFarland, Hinterberger, Birbaumer & Wolpaw (2004), IEEE Trans Biomed Eng 51(6), DOI 10.1109/TBME.2004.827072; dataset DOI 10.13026/C28G6P (PhysioNet v1.0.0). From data/directory.yaml: BCI2000 with a 64-channel 10-10 cap, 160 Hz, no online filters, 60 Hz mains, 109 volunteers with undocumented demographics, access: open, licence ODC-By 1.0. Caveats the directory records and this notebook acts on: no hardware filters, so line noise and drift are present; S088, S089, S092 and S100 carry inconsistent event timestamps and S038 and S104 are also often dropped; channel names need mapping for a montage.

Every EDF is deleted in a finally; free disk is printed before and after.

In [1]:
# Setup: dependencies, the shared helpers, non-interactive plotting.
import importlib.util
import subprocess
import sys
import warnings
from pathlib import Path

# 1. Dependencies are pinned in notebooks/requirements.txt.  Nothing is installed when the
#    pinned stack is already present (local runs, CI); a fresh Colab or Binder kernel installs
#    it once.  On Colab, run from a clone of the repository so that notebooks/_shared/ is
#    available (repository URL: TODO(confirm), spec section 13 item 3).
_needed = ('mne', 'scipy', 'matplotlib', 'pooch', 'specparam')
_missing = [p for p in _needed if importlib.util.find_spec(p) is None]
if _missing:
    _req = next((d / "requirements.txt" for d in (Path.cwd(), *Path.cwd().parents)
                 if (d / "requirements.txt").exists()), None)
    _cmd = [sys.executable, "-m", "pip", "install", "-q"]
    _cmd += ["-r", str(_req)] if _req else ["mne==1.10.2", "pooch>=1.8"]
    subprocess.check_call(_cmd)

# 2. Shared helpers, located relative to the working directory -- notebooks/<level>/ or
#    notebooks/ -- never through an absolute path.
_shared = next((d / "_shared" for d in (Path.cwd(), *Path.cwd().parents)
                if (d / "_shared" / "helpers_l4.py").exists()), None)
if _shared is None:
    raise FileNotFoundError("start the kernel in notebooks/L4/ (or notebooks/) so that "
                            "_shared/helpers_l4.py is found")
sys.path.insert(0, str(_shared))
import helpers
import helpers_l1
import helpers_l4 as L4
from scipy import signal as sps
from scipy import stats as sstats

# 3. Plotting: Jupyter's default inline backend renders static PNGs through Agg (no windows,
#    nothing blocks); outside Jupyter the helpers select Agg.  Every MNE figure is requested
#    with show=False and each figure cell ends with plt.show().
import matplotlib.pyplot as plt
import numpy as np
import mne

mne.set_log_level("WARNING")
plt.rcParams["figure.dpi"] = 72
print(f"MNE {mne.__version__}; helpers_l4 imported from notebooks/_shared")
print(f"downloads go to {helpers_l1.download_dir().name}/ (resolved relative to the working directory, "
      "or $EEG_COURSE_DOWNLOADS) and are deleted at the end of this notebook")
MNE 1.10.2; helpers_l4 imported from notebooks/_shared
downloads go to eeg-course/ (resolved relative to the working directory, or $EEG_COURSE_DOWNLOADS) and are deleted at the end of this notebook

1 · The plan, fixed before the data

Written down first so that nothing below is a choice made after looking.

  • Subset. FULL_COHORT = Falsehelpers_l4.SUBSET_DEFAULT, the first ten subjects that survive the exclusion rules. The rules themselves come from data/directory.yaml, not from this notebook.
  • Pipeline. helpers_l4.MI_PIPELINE, unchanged from every other Level-4 notebook: average reference, no filtering, no epoch rejection, epochs −2.5…4.5 s, Morlet 4–40 Hz at max(3, f/2) cycles.
  • Baseline −1.5…−0.5 s, chosen because it lies entirely outside the edge region (the longest wavelet reaches 0.597 s in from each end, so the edge ends at −1.903 s) and stops 0.5 s before the cue. It was not moved after looking at any map.
  • Active window 0.5…3.5 s, inside the 4.1 s task and clear of the far edge.
  • Normalisation: percent change, band-first order — average raw power over trials, then over the band, then normalise. nb-4-4-erd shows that the other orders give different answers, so the order is part of the method.
  • Bands mu 8–13 Hz and beta 13–30 Hz, fixed in helpers_l4.
  • Statistic the lateralisation index C3 − C4 in percentage points, contrasted between the two conditions, tested with a cluster-permutation test over the edge-safe frequency × time grid, plus one a-priori ROI.
  • Peak check specparam on each subject's pre-cue baseline at C3 and C4; a subject with no fitted peak in 7–13 Hz is reported, not dropped.
In [2]:
FULL_COHORT = False          # True: every loadable subject (~103, ~740 MB) -- a local run, not CI

CHANNELS = ["C3", "C4"]
if FULL_COHORT:
    SUBJECTS = [s for s in (f"S{i:03d}" for i in range(1, 110))
                if s not in helpers.EEGBCI_EXCLUDE_DEFAULT and s not in helpers.EEGBCI_EXCLUDE_OPTIONAL]
else:
    SUBJECTS = list(L4.SUBSET_DEFAULT)
print(f"FULL_COHORT = {FULL_COHORT} -> {len(SUBJECTS)} subjects: {SUBJECTS[0]}..{SUBJECTS[-1]}")
print(f"excluded always : {', '.join(helpers.EEGBCI_EXCLUDE_DEFAULT)} -- "
      f"'{L4.DATASETS_L4['ds-eegbci']['caveats'][1]}'")
print(f"excluded by default: {', '.join(helpers.EEGBCI_EXCLUDE_OPTIONAL)} (same caveat)")
print(f"download estimate: {len(SUBJECTS) * len(L4.MI_RUNS) * 2.4:.0f} MB, deleted at the end")
for key, value in L4.MI_PIPELINE.items():
    print(f"  {key:16s} : {value}")
FULL_COHORT = False -> 10 subjects: S001..S010
excluded always : S088, S089, S092, S100 -- 'Subjects S088, S089, S092 and S100 carry inconsistent event timestamps; S038 and S104 are also often dropped.'
excluded by default: S038, S104 (same caveat)
download estimate: 72 MB, deleted at the end
  dataset          : ds-eegbci, motor-imagery runs R04 + R08 + R12 (left or right fist, imagined)
  conditions       : T1 = left-fist imagery, T2 = right-fist imagery, read from the EDF's own annotations; T0 (rest) is not epoched
  excluded_subjects : S088, S089, S092, S100 always (inconsistent event timestamps, data/directory.yaml) and S038, S104 by default (several reports additionally drop them); helpers.load_spine refuses them rather than this module filtering them
  channel_names    : mne.datasets.eegbci.standardize (Fc5. -> FC5 etc.), then the standard_1005 montage matched by name
  reference        : average of the 64 EEG channels, applied offline.  The recording's own reference is TODO(confirm) in data/directory.yaml, so an explicit one is set before any comparison
  filter           : none.  The EDF is read at its native 160 Hz and nothing is resampled or filtered: the Morlet transform is the band-pass, and a prior band-pass would only narrow it further
  epochs           : -2.5 to 4.5 s around the cue, NO baseline correction (a time-frequency baseline is applied to power, not to the signal), no annotation-based rejection
  rejection        : none by default.  Rejecting on amplitude before a power measurement biases the power, and the cohort notebooks report each subject's largest absolute sample instead so that a reader can see what was kept (label_source: algorithmic)
  time_frequency   : mne.time_frequency.tfr_array_morlet on the single trials, freqs = 4..40 Hz in 1 Hz steps, n_cycles = max(3, f/2), zero_mean=True, use_fft=True, decim=4; output='power', so what is averaged is single-trial (total) power
  normalisation    : percent change from a -1.5..-0.5 s baseline, with the averaging order stated every time it is reported (helpers_l4.ERD_ORDERS)
  measurement      : mu = 8-13 Hz and beta = 13-30 Hz averaged over 0.5..3.5 s at C3 and C4

2 · One pass over the cohort

Each subject is loaded, transformed, summarised and its EDFs are deleted before the next subject is loaded, so the peak disk cost is one subject rather than the whole cohort. A subject that fails to load is recorded as a row rather than stopping the run.

In [3]:
import time

dl = L4.Downloads("nb-c4").start()
t0 = time.time()
per_subject, failures = {}, []
for sid in SUBJECTS:
    try:
        ep, nfo = L4.load_imagery_epochs(sid, L4.MI_RUNS, dl)
        rec = {"info": nfo}
        for cond in ("T1", "T2"):
            if cond not in ep.event_id or len(ep[cond]) == 0:
                raise ValueError(f"no {cond} epochs")
            X = ep[cond].get_data(picks=CHANNELS) * 1e6
            r = L4.morlet_power(X, sfreq=float(ep.info["sfreq"]), times=ep.times)
            rec[cond] = r["power"]                      # (n_trials, 2, n_freqs, n_times)
        # pre-cue baseline spectrum for the peak check, from the same epochs, C3 and C4
        base = ep.copy().crop(tmin=L4.MI_TMIN, tmax=0.0).get_data(picks=CHANNELS) * 1e6
        rec["baseline_segments"] = base.reshape(-1, base.shape[-1])
        rec["sfreq"] = float(ep.info["sfreq"])
        per_subject[sid] = rec
        freqs, times = r["freqs"], r["times"]
    except Exception as exc:
        failures.append({"subject": sid, "reason": f"{type(exc).__name__}: {exc}"})
        print(f"  {sid}: FAILED -- {type(exc).__name__}: {exc}")
    finally:
        dl.cleanup()                                    # this subject's EDFs go before the next one arrives
COHORT_S = time.time() - t0
edge = L4.edge_seconds(freqs, L4.TF_CYCLES)
safe = (times >= times[0] + edge.max()) & (times <= times[-1] - edge.max())
print(f"\n{len(per_subject)} of {len(SUBJECTS)} subjects processed in {COHORT_S:.0f} s "
      f"({COHORT_S / max(len(per_subject), 1):.1f} s each); {len(failures)} failure(s)")
print(f"edge-safe window {times[safe][0]:+.3f}..{times[safe][-1]:+.3f} s "
      f"({safe.sum()} of {len(times)} points); baseline {L4.TF_BASELINE[0]:g}..{L4.TF_BASELINE[1]:g} s ends "
      f"{L4.TF_BASELINE[0] - (times[0] + edge.max()):+.3f} s after the edge region, so no baseline sample is "
      f"made of zero padding")
free disk before nb-c4: 4,378 MB
deleted 3 downloaded file(s), 7.4 MiB freed
deleted 3 downloaded file(s), 7.3 MiB freed
deleted 3 downloaded file(s), 7.4 MiB freed
deleted 3 downloaded file(s), 7.3 MiB freed
deleted 3 downloaded file(s), 7.3 MiB freed
deleted 3 downloaded file(s), 7.3 MiB freed
deleted 3 downloaded file(s), 7.4 MiB freed
deleted 3 downloaded file(s), 7.3 MiB freed
deleted 3 downloaded file(s), 7.3 MiB freed
deleted 3 downloaded file(s), 7.3 MiB freed

10 of 10 subjects processed in 835 s (83.5 s each); 0 failure(s)
edge-safe window -1.900..+3.900 s (233 of 281 points); baseline -1.5..-0.5 s ends +0.403 s after the edge region, so no baseline sample is made of zero padding
In [4]:
rows = []
lat = {}
for sid, rec in per_subject.items():
    row = {"subject": sid, "T1 trials": rec["info"]["n_epochs"].get("T1", 0),
           "T2 trials": rec["info"]["n_epochs"].get("T2", 0),
           "max |sample| (uV)": max(rec["info"]["max_abs_uv"].values())}
    pc = {}
    for cond in ("T1", "T2"):
        P = rec[cond].mean(0)                            # (2, n_freqs, n_times) raw, trial-averaged
        b = (times >= L4.TF_BASELINE[0]) & (times <= L4.TF_BASELINE[1])
        pc[cond] = np.stack([100 * (P[k] - P[k][:, b].mean(1, keepdims=True))
                             / P[k][:, b].mean(1, keepdims=True) for k in range(len(CHANNELS))])
        for band_name, band in (("mu", L4.MU_BAND), ("beta", L4.BETA_BAND)):
            for k, ch in enumerate(CHANNELS):
                row[f"{cond} {band_name} {ch} (%)"] = L4.band_erd(
                    rec[cond][:, k], freqs, times, band, baseline=L4.TF_BASELINE, active=L4.TF_ACTIVE,
                    order="band-first")
    lat[sid] = {c: pc[c][0] - pc[c][1] for c in ("T1", "T2")}
    row["T2 contralateral (C3 < C4)"] = row["T2 mu C3 (%)"] < row["T2 mu C4 (%)"]
    row["T1 contralateral (C4 < C3)"] = row["T1 mu C4 (%)"] < row["T1 mu C3 (%)"]
    rows.append(row)
cols = ["subject", "T1 trials", "T2 trials", "max |sample| (uV)",
        "T2 mu C3 (%)", "T2 mu C4 (%)", "T1 mu C3 (%)", "T1 mu C4 (%)",
        "T2 beta C3 (%)", "T2 beta C4 (%)", "T1 beta C3 (%)", "T1 beta C4 (%)",
        "T2 contralateral (C3 < C4)", "T1 contralateral (C4 < C3)"]
print(f"Per subject: percent change from baseline, band-first order, "
      f"{L4.TF_ACTIVE[0]:g}..{L4.TF_ACTIVE[1]:g} s against {L4.TF_BASELINE[0]:g}..{L4.TF_BASELINE[1]:g} s")
print(L4.fmt_table(rows, cols, floatfmt="{:.1f}"))
n_t2 = sum(r["T2 contralateral (C3 < C4)"] for r in rows)
n_t1 = sum(r["T1 contralateral (C4 < C3)"] for r in rows)
print()
print(f"Contralateral dominance in the mu band: {n_t2} of {len(rows)} subjects for right-hand imagery "
      f"(C3 more negative than C4), {n_t1} of {len(rows)} for left-hand imagery.")
print(f"Binomial test against chance (p = .5): right-hand p = "
      f"{sstats.binomtest(n_t2, len(rows), 0.5).pvalue:.4f}, left-hand p = "
      f"{sstats.binomtest(n_t1, len(rows), 0.5).pvalue:.4f}")
Per subject: percent change from baseline, band-first order, 0.5..3.5 s against -1.5..-0.5 s
subject  T1 trials  T2 trials  max |sample| (uV)  T2 mu C3 (%)  T2 mu C4 (%)  T1 mu C3 (%)  T1 mu C4 (%)  T2 beta C3 (%)  T2 beta C4 (%)  T1 beta C3 (%)  T1 beta C4 (%)  T2 contralateral (C3 < C4)  T1 contralateral (C4 < C3)
-------  ---------  ---------  -----------------  ------------  ------------  ------------  ------------  --------------  --------------  --------------  --------------  --------------------------  --------------------------
   S001         23         22              511.2         -47.9         -10.5         -29.7         -20.5           -35.8           -12.0           -19.0           -10.6                        True                       False
   S002         22         22              282.7         -32.7         -16.3         -12.0          11.0           -36.6           -15.5           -15.7           -16.2                        True                       False
   S003         23         22              539.0          -5.3         -12.6         -17.6          -2.3            -6.9            -3.8            -3.0            -3.0                       False                       False
   S004         22         22              419.1         -38.3          -8.4         -35.7         -32.8           -20.2             9.4           -16.6            -2.5                        True                       False
   S005         21         23              317.8           1.8          -4.9          -4.4           7.8           -29.0           -19.9            -8.6           -14.7                       False                       False
   S006         23         21              255.0         -12.2         -12.8           0.2         -18.8           -30.6            -9.1            -8.1            -9.1                       False                        True
   S007         23         22              377.6         -21.3           1.6           8.8         -37.0           -37.7           -20.4           -15.7           -37.7                        True                        True
   S008         21         23              433.6         -41.1         -30.2         -34.6         -41.5           -12.1           -18.2           -18.9           -24.8                        True                        True
   S009         23         21              756.6           6.6         -12.3          13.5          -8.5            11.7             9.2            12.6            20.9                       False                        True
   S010         23         21              644.4         -24.9          -9.4         -22.3         -42.2           -21.7           -20.5           -27.9           -23.0                        True                        True

Contralateral dominance in the mu band: 6 of 10 subjects for right-hand imagery (C3 more negative than C4), 5 of 10 for left-hand imagery.
Binomial test against chance (p = .5): right-hand p = 0.7539, left-hand p = 1.0000

3 · Grand-average maps and the lateralisation

Deliverable 1: the ERD maps. Deliverable 2: the lateralisation at C3 and C4.

In [5]:
fig, axes = plt.subplots(2, 3, figsize=(17.5, 8.0))
G = {c: np.mean([np.stack([100 * (per_subject[s][c].mean(0)[k]
                                  - per_subject[s][c].mean(0)[k][:, (times >= L4.TF_BASELINE[0])
                                                                & (times <= L4.TF_BASELINE[1])].mean(1, keepdims=True))
                           / per_subject[s][c].mean(0)[k][:, (times >= L4.TF_BASELINE[0])
                                                          & (times <= L4.TF_BASELINE[1])].mean(1, keepdims=True)
                           for k in range(2)]) for s in per_subject], axis=0)
     for c in ("T1", "T2")}
for r, cond in enumerate(("T2", "T1")):
    for k, ch in enumerate(CHANNELS):
        L4.plot_tfr(G[cond][k], freqs, times, ax=axes[r, k], vlim=(-40, 40), edge_s=edge,
                    baseline=L4.TF_BASELINE, cbar_label="Power change from baseline (%)",
                    title=f"{cond} ({L4.MI_CONDITIONS[cond]}) at {ch}\ngrand average, "
                          f"{len(per_subject)} subjects (%)")
    L4.plot_tfr(G[cond][0] - G[cond][1], freqs, times, ax=axes[r, 2], vlim=(-25, 25), edge_s=edge,
                baseline=L4.TF_BASELINE, cbar_label="C3 - C4 (percentage points)",
                title=f"{cond}: C3 - C4 lateralisation\n(percentage points)")
fig.suptitle(f"Capstone C4 deliverable 1-2: ERD maps and lateralisation, {len(per_subject)} ds-eegbci subjects",
             y=1.01)
fig.tight_layout()
plt.show()   # render the static figure(s) of this cell inline
Figure 1 of notebook nb-c4-mu-beta-erd, an output plot. The text around it states what it shows and the units of every axis.

4 · Does each subject have a mu peak?

Band power in 8–13 Hz can be computed for anybody. It only describes a rhythm if there is a peak over the aperiodic background, so specparam is run on each subject's pre-cue data at C3 and C4 — pre-cue, because a peak measured on the task window would be measured on the thing being tested.

A subject with no fitted peak in 7–13 Hz is reported as such. Their ERD number is still computed and still printed; what changes is what it can be called.

In [6]:
peaks = []
for sid, rec in per_subject.items():
    sf_s = rec["sfreq"]
    segs = rec["baseline_segments"]
    f_w, P_w = sps.welch(segs, fs=sf_s, nperseg=int(min(2 * sf_s, segs.shape[-1])),
                         noverlap=int(min(sf_s, segs.shape[-1] // 2)), window="hann", axis=-1)
    m = (f_w >= 1.0) & (f_w <= 40.0)
    row = {"subject": sid}
    for k, ch in enumerate(CHANNELS):
        psd = P_w[k::len(CHANNELS)].mean(0)
        fit = helpers_l1.fit_specparam(f_w[m], psd[m])
        row[f"{ch} peak (Hz)"] = fit["iaf"]
        row[f"{ch} exponent"] = fit["exponent"]
        row[f"{ch} r^2"] = fit["r_squared"]
    row["has a mu peak"] = bool(np.isfinite(row["C3 peak (Hz)"]) or np.isfinite(row["C4 peak (Hz)"]))
    peaks.append(row)
print(f"specparam on the pre-cue window ({L4.MI_TMIN:g}..0 s of every epoch, both conditions pooled), "
      f"fit range {helpers_l1.FIT_RANGE[0]:g}-{helpers_l1.FIT_RANGE[1]:g} Hz, peak search "
      f"{helpers_l1.IAF_RANGE[0]:g}-{helpers_l1.IAF_RANGE[1]:g} Hz:")
print(L4.fmt_table(peaks, floatfmt="{:.3f}"))
NO_PEAK = [r["subject"] for r in peaks if not r["has a mu peak"]]
print()
print(f"Subjects with NO fitted mu peak at either C3 or C4: "
      + (", ".join(NO_PEAK) if NO_PEAK else "none of them")
      + f"  ({len(NO_PEAK)} of {len(peaks)})")
print("For those subjects the 8-13 Hz band power is a measurement of the aperiodic background. The ERD "
      "number is still meaningful as a change in band power; calling it 'mu desynchronisation' is the part "
      "that is not supported.")

fig, axes = plt.subplots(1, 2, figsize=(13.5, 4.2))
for ax, ch in zip(axes, CHANNELS):
    v = np.array([r[f"{ch} peak (Hz)"] for r in peaks])
    e = np.array([r[f"{ch} exponent"] for r in peaks])
    ok = np.isfinite(v)
    ax.scatter(v[ok], e[ok], s=40, label=f"{int(ok.sum())} with a peak")
    for i, r in enumerate(peaks):
        if ok[i]:
            ax.annotate(r["subject"][-3:], (v[i], e[i]), fontsize=7, xytext=(3, 3),
                        textcoords="offset points")
    if (~ok).any():
        ax.scatter(np.full((~ok).sum(), helpers_l1.IAF_RANGE[0] - 0.4), e[~ok], marker="x", color="tab:red",
                   s=50, label=f"{int((~ok).sum())} with NO peak (plotted at the left edge)")
    ax.set(xlabel="Fitted alpha/mu peak frequency (Hz)", ylabel="Aperiodic exponent (dimensionless)",
           title=f"{ch}, pre-cue baseline: peak frequency against aperiodic exponent\n"
                 f"{len(peaks)} subjects (Hz and dimensionless)")
    ax.legend(fontsize=8)
    ax.grid(alpha=0.3)
fig.tight_layout()
plt.show()   # render the static figure(s) of this cell inline
specparam on the pre-cue window (-2.5..0 s of every epoch, both conditions pooled), fit range 1-40 Hz, peak search 7-13 Hz:
subject  C3 peak (Hz)  C3 exponent  C3 r^2  C4 peak (Hz)  C4 exponent  C4 r^2  has a mu peak
-------  ------------  -----------  ------  ------------  -----------  ------  -------------
   S001        12.707        1.255   0.993        12.400        1.331   0.992           True
   S002        11.846        0.791   0.857        11.839        0.885   0.988           True
   S003         7.452        1.321   0.944           nan        1.301   0.986           True
   S004           nan        0.548   0.958           nan        0.563   0.958          False
   S005           nan        0.896   0.983           nan        0.979   0.983          False
   S006           nan        0.283   0.915         8.891        0.742   0.914           True
   S007        12.359        1.195   0.994        12.536        1.303   0.991           True
   S008        10.411        1.089   0.957        10.310        1.169   0.972           True
   S009           nan        0.510   0.911           nan        0.465   0.583          False
   S010        11.768        1.614   0.988        11.355        1.713   0.990           True

Subjects with NO fitted mu peak at either C3 or C4: S004, S005, S009  (3 of 10)
For those subjects the 8-13 Hz band power is a measurement of the aperiodic background. The ERD number is still meaningful as a change in band power; calling it 'mu desynchronisation' is the part that is not supported.
Figure 2 of notebook nb-c4-mu-beta-erd, an output plot. The text around it states what it shows and the units of every axis.

5 · The cluster test on the difference

The subject-level quantity is the lateralisation map C3 − C4 in percentage points; the contrast is that map for left-hand imagery minus the same map for right-hand imagery, so a subject who simply desynchronises more overall contributes nothing to it. With n subjects there are 2ⁿ distinct sign flips, so for a subset of ten the permutation distribution is exact at 1024.

What a significant cluster licenses. That the null of no difference anywhere in the tested window is rejected. It does not license a claim about when the effect starts, when it ends, or which frequency carries it — the cluster's extent is a property of the cluster-forming threshold as much as of the data (pf-cluster-inference-misread). The a-priori ROI beside it is the statement that can be made about a particular band and window, because the band and window were chosen before the data.

In [7]:
X_diff = np.stack([(lat[s]["T1"] - lat[s]["T2"])[:, safe] for s in per_subject])
n_sub = X_diff.shape[0]
N_PERM = min(2 ** n_sub, 10000)
adjacency = mne.stats.combine_adjacency(len(freqs), int(safe.sum()))
thr = sstats.t.ppf(1 - 0.05 / 2, n_sub - 1)
with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    T, clusters, cluster_p, H0 = mne.stats.permutation_cluster_1samp_test(
        X_diff, threshold=thr, n_permutations=N_PERM, tail=0, adjacency=adjacency, out_type="mask",
        seed=20260918, verbose=False)
print(f"Cluster-permutation test: {n_sub} subjects, {len(freqs)} x {int(safe.sum())} cells, "
      f"cluster-forming |t| > {thr:.3f} (two-tailed p < .05), {N_PERM} permutations "
      f"({'exact -- all 2^n sign flips' if N_PERM == 2 ** n_sub else 'sampled'})")
print(f"  {len(clusters)} candidate clusters; {sum(p < 0.05 for p in cluster_p)} at p < .05")
for i, p in enumerate(cluster_p):
    if p < 0.10:
        f_i, t_i = np.where(clusters[i])
        print(f"    cluster {i}: p = {p:.4f}, {int(clusters[i].sum())} cells, "
              f"{freqs[f_i.min()]:.0f}-{freqs[f_i.max()]:.0f} Hz, "
              f"{times[safe][t_i.min()]:+.2f}..{times[safe][t_i.max()]:+.2f} s, sum t = {T[clusters[i]].sum():+.1f}")
if not len(cluster_p):
    print("    no cell passed the cluster-forming threshold, so there is nothing to test")

fm = (freqs >= L4.MU_BAND[0]) & (freqs <= L4.MU_BAND[1])
tm = (times[safe] >= L4.TF_ACTIVE[0]) & (times[safe] <= L4.TF_ACTIVE[1])
roi = X_diff[:, fm][:, :, tm].mean(axis=(1, 2))
t_roi, p_roi = sstats.ttest_1samp(roi, 0)
print()
print(f"A-priori ROI (mu {L4.MU_BAND[0]:g}-{L4.MU_BAND[1]:g} Hz, {L4.TF_ACTIVE[0]:g}..{L4.TF_ACTIVE[1]:g} s), "
      f"one test, no correction needed:")
print(f"  mean {roi.mean():+.3f} percentage points, SD {roi.std(ddof=1):.3f}, "
      f"t({n_sub - 1}) = {t_roi:+.3f}, p = {p_roi:.4f}, dz = {roi.mean() / roi.std(ddof=1):+.3f}; "
      f"{int((roi > 0).sum())} of {n_sub} subjects with the expected sign")

fig, axes = plt.subplots(1, 2, figsize=(14.5, 4.2))
L4.plot_tf_clusters(T, clusters, cluster_p, freqs, times[safe], ax=axes[0],
                    title=f"Left minus right imagery, (C3 - C4) lateralisation\n{n_sub} subjects, "
                          f"significant clusters outlined (t)")
axes[1].hist(H0, bins=30, color="0.7", label=f"{N_PERM} sign-flip permutations")
if len(cluster_p):
    best = int(np.argmin(cluster_p))
    axes[1].axvline(T[clusters[best]].sum(), color="tab:red", lw=2,
                    label=f"largest observed cluster, sum t = {T[clusters[best]].sum():+.1f}, "
                          f"p = {cluster_p[best]:.4f}")
axes[1].set(xlabel="Largest cluster sum-t under the null (t units)", ylabel="Permutations",
            title="The permutation null distribution (t units)")
axes[1].legend(fontsize=8)
fig.tight_layout()
plt.show()   # render the static figure(s) of this cell inline
Cluster-permutation test: 10 subjects, 37 x 233 cells, cluster-forming |t| > 2.262 (two-tailed p < .05), 1024 permutations (exact -- all 2^n sign flips)
  55 candidate clusters; 0 at p < .05
    cluster 5: p = 0.0859, 72 cells, 8-15 Hz, +0.47..+0.80 s, sum t = +206.1

A-priori ROI (mu 8-13 Hz, 0.5..3.5 s), one test, no correction needed:
  mean +15.257 percentage points, SD 25.968, t(9) = +1.858, p = 0.0961, dz = +0.588; 7 of 10 subjects with the expected sign
Figure 3 of notebook nb-c4-mu-beta-erd, an output plot. The text around it states what it shows and the units of every axis.

6 · Deliverables and rubric

In [8]:
try:
    print("nb-c4-mu-beta-erd -- Capstone C4 deliverables (draft; TODO(confirm) at author review)")
    print(f"Dataset: ds-eegbci (EEGMMIDB), PhysioNet DOI {L4.DATASETS_L4['ds-eegbci']['dataset_doi']}, "
          f"licence {L4.DATASETS_L4['ds-eegbci']['license']}, access {L4.DATASETS_L4['ds-eegbci']['access']}.")
    print(f"Cohort: FULL_COHORT = {FULL_COHORT} -> {len(per_subject)} of {len(SUBJECTS)} subjects processed "
          f"({len(failures)} failure(s)); runs {'+'.join(f'R{r:02d}' for r in L4.MI_RUNS)}.")
    print(f"Excluded with the reason recorded: {', '.join(helpers.EEGBCI_EXCLUDE_DEFAULT)} "
          f"(inconsistent event timestamps, data/directory.yaml) and "
          f"{', '.join(helpers.EEGBCI_EXCLUDE_OPTIONAL)} (also often dropped).")
    print(f"Wall clock: {COHORT_S:.0f} s for {len(per_subject)} subjects "
          f"({COHORT_S / max(len(per_subject), 1):.1f} s each), downloads deleted per subject.")
    print()
    print("DELIVERABLE 1 -- ERD maps, left vs right hand imagery, per subject (band-first percent change, "
          f"baseline {L4.TF_BASELINE[0]:g}..{L4.TF_BASELINE[1]:g} s, window {L4.TF_ACTIVE[0]:g}.."
          f"{L4.TF_ACTIVE[1]:g} s):")
    for r in rows:
        print(f"    {r['subject']}  T2 mu C3 {r['T2 mu C3 (%)']:+7.1f} % C4 {r['T2 mu C4 (%)']:+7.1f} % | "
              f"T1 mu C3 {r['T1 mu C3 (%)']:+7.1f} % C4 {r['T1 mu C4 (%)']:+7.1f} % | "
              f"T2 beta C3 {r['T2 beta C3 (%)']:+7.1f} % C4 {r['T2 beta C4 (%)']:+7.1f} %")
    mu_c3 = np.array([r["T2 mu C3 (%)"] for r in rows])
    mu_c4 = np.array([r["T2 mu C4 (%)"] for r in rows])
    print(f"    cohort median, right-hand imagery: C3 {np.median(mu_c3):+.1f} % "
          f"(range {mu_c3.min():+.1f} to {mu_c3.max():+.1f}), C4 {np.median(mu_c4):+.1f} % "
          f"(range {mu_c4.min():+.1f} to {mu_c4.max():+.1f})")
    print(f"    NOTE: nb-4-4-erd quotes S001 alone at C3 -47.9 % / C4 -10.5 %. S001 was CHOSEN for the size "
          f"of its effect, and the cohort median above is where a typical subject sits. Quoting the one "
          f"without the other is the selection effect Level 6 is about.")
    print()
    print(f"DELIVERABLE 2 -- contralateral lateralisation: {n_t2} of {len(rows)} subjects show it for "
          f"right-hand imagery (binomial p = {sstats.binomtest(n_t2, len(rows), 0.5).pvalue:.4f}) and "
          f"{n_t1} of {len(rows)} for left-hand imagery "
          f"(p = {sstats.binomtest(n_t1, len(rows), 0.5).pvalue:.4f}).")
    print()
    print("DELIVERABLE 3 -- cluster test on the difference:")
    print(f"    {len(clusters)} candidate clusters, {sum(p < 0.05 for p in cluster_p)} at p < .05, "
          f"{N_PERM} permutations (exact for {n_sub} subjects), cluster-forming |t| > {thr:.3f}.")
    for i, p in enumerate(cluster_p):
        if p < 0.10:
            f_i, t_i = np.where(clusters[i])
            print(f"        cluster {i}: p = {p:.4f}, {int(clusters[i].sum())} cells, "
                  f"{freqs[f_i.min()]:.0f}-{freqs[f_i.max()]:.0f} Hz, "
                  f"{times[safe][t_i.min()]:+.2f}..{times[safe][t_i.max()]:+.2f} s")
    print(f"    A-priori ROI (one test, chosen before the data): t({n_sub - 1}) = {t_roi:+.3f}, "
          f"p = {p_roi:.4f}, mean {roi.mean():+.3f} percentage points, dz {roi.mean() / roi.std(ddof=1):+.3f}.")
    print(f"    INTERPRETATION: a significant cluster rejects the null of no difference ANYWHERE in the "
          f"tested window. It does not locate the effect in time or frequency, and the cluster's extent "
          f"would move with the cluster-forming threshold (nb-4-7-tf-cluster shows by how much).")
    print()
    print("DELIVERABLE 4 -- who has a mu peak:")
    for r in peaks:
        c3 = f"{r['C3 peak (Hz)']:.2f}" if np.isfinite(r["C3 peak (Hz)"]) else "  none"
        c4 = f"{r['C4 peak (Hz)']:.2f}" if np.isfinite(r["C4 peak (Hz)"]) else "  none"
        print(f"    {r['subject']}  C3 peak {c3:>6s} Hz (exponent {r['C3 exponent']:.2f}, r^2 "
              f"{r['C3 r^2']:.3f}) | C4 peak {c4:>6s} Hz (exponent {r['C4 exponent']:.2f}, r^2 "
              f"{r['C4 r^2']:.3f})" + ("   <-- NO MU PEAK AT EITHER SITE" if not r["has a mu peak"] else ""))
    print(f"    {len(NO_PEAK)} of {len(peaks)} subjects have no fitted peak in "
          f"{helpers_l1.IAF_RANGE[0]:g}-{helpers_l1.IAF_RANGE[1]:g} Hz at either site"
          + (f": {', '.join(NO_PEAK)}" if NO_PEAK else "."))
    print()
    RUBRIC = [
        ("Baseline choice justified and edge-safe",
         f"{L4.TF_BASELINE[0]:g}..{L4.TF_BASELINE[1]:g} s, fixed in helpers_l4.TF_BASELINE before the data; "
         f"the edge region ends at {times[0] + edge.max():+.3f} s, so the baseline starts "
         f"{L4.TF_BASELINE[0] - (times[0] + edge.max()):.3f} s clear of it and ends 0.5 s before the cue. "
         f"Every map shades the edge region."),
        ("Normalisation stated",
         "percent change from the baseline mean, per frequency, in the BAND-FIRST order (raw power averaged "
         "over trials, then over the band, then normalised). nb-4-4-erd shows the three other orders give "
         "different answers, so the order is part of the method and is stated wherever a number is."),
        ("Cluster interpretation correct",
         f"the test rejects the null of no difference anywhere in "
         f"{freqs[0]:.0f}-{freqs[-1]:.0f} Hz x {times[safe][0]:+.2f}..{times[safe][-1]:+.2f} s; no claim is "
         f"made about the effect's onset, offset or frequency, and the a-priori ROI is reported beside it as "
         f"the statement that can be made about a named band and window."),
        ("Subjects without a mu peak handled explicitly",
         f"specparam is fitted on each subject's PRE-CUE data at C3 and C4; "
         f"{len(NO_PEAK)} of {len(peaks)} subjects have no peak in "
         f"{helpers_l1.IAF_RANGE[0]:g}-{helpers_l1.IAF_RANGE[1]:g} Hz"
         + (f" ({', '.join(NO_PEAK)})" if NO_PEAK else "")
         + ". They are reported, not dropped; their band-power change is still a change in band power, but "
           "it is not evidence about a rhythm."),
        ("Documented defective subjects excluded with the reason recorded",
         f"{', '.join(helpers.EEGBCI_EXCLUDE_DEFAULT)} always and "
         f"{', '.join(helpers.EEGBCI_EXCLUDE_OPTIONAL)} by default, refused by helpers.load_spine with the "
         f"data/directory.yaml caveat as the reason; none is in this subset."),
        ("Runs on a documented subset within the section 11 limit, with a FULL_COHORT switch",
         f"FULL_COHORT = {FULL_COHORT}; {len(per_subject)} subjects in {COHORT_S:.0f} s here. "
         f"FULL_COHORT = True is about 103 subjects and ~740 MB and has NOT been executed."),
    ]
    print("RUBRIC:")
    for item, state in RUBRIC:
        print(f"  [x] {item}\n      {state}")
    if failures:
        print()
        print("Failures recorded rather than raised:")
        for f_ in failures:
            print(f"    {f_['subject']}: {f_['reason']}")
    print()
    print("Pitfalls: pf-tf-edge-effects, pf-band-power-slope, pf-cluster-inference-misread, "
          "pf-uncorrected-timepoint-tests. Widgets: w-tf-baseline-explorer, w-burst-detector, "
          "w-cluster-permutation-viz (mode tf), w-aperiodic-explorer.")
    print("TODO(confirm): the catalog carries no published ERD value or lateralisation effect size for this "
          "dataset, so nothing above is compared with a literature number; and this dataset's reference is "
          "recorded as TODO(confirm) in data/directory.yaml, so the average reference used here is a choice "
          "of this course, stated rather than inherited.")
finally:
    dl.finish()
nb-c4-mu-beta-erd -- Capstone C4 deliverables (draft; TODO(confirm) at author review)
Dataset: ds-eegbci (EEGMMIDB), PhysioNet DOI 10.13026/C28G6P, licence ODC-By-1.0 (data/directory.yaml; ODC-By 1.0 on PhysioNet, CC0 on the OpenNeuro BIDS mirror ds004362), access open.
Cohort: FULL_COHORT = False -> 10 of 10 subjects processed (0 failure(s)); runs R04+R08+R12.
Excluded with the reason recorded: S088, S089, S092, S100 (inconsistent event timestamps, data/directory.yaml) and S038, S104 (also often dropped).
Wall clock: 835 s for 10 subjects (83.5 s each), downloads deleted per subject.

DELIVERABLE 1 -- ERD maps, left vs right hand imagery, per subject (band-first percent change, baseline -1.5..-0.5 s, window 0.5..3.5 s):
    S001  T2 mu C3   -47.9 % C4   -10.5 % | T1 mu C3   -29.7 % C4   -20.5 % | T2 beta C3   -35.8 % C4   -12.0 %
    S002  T2 mu C3   -32.7 % C4   -16.3 % | T1 mu C3   -12.0 % C4   +11.0 % | T2 beta C3   -36.6 % C4   -15.5 %
    S003  T2 mu C3    -5.3 % C4   -12.6 % | T1 mu C3   -17.6 % C4    -2.3 % | T2 beta C3    -6.9 % C4    -3.8 %
    S004  T2 mu C3   -38.3 % C4    -8.4 % | T1 mu C3   -35.7 % C4   -32.8 % | T2 beta C3   -20.2 % C4    +9.4 %
    S005  T2 mu C3    +1.8 % C4    -4.9 % | T1 mu C3    -4.4 % C4    +7.8 % | T2 beta C3   -29.0 % C4   -19.9 %
    S006  T2 mu C3   -12.2 % C4   -12.8 % | T1 mu C3    +0.2 % C4   -18.8 % | T2 beta C3   -30.6 % C4    -9.1 %
    S007  T2 mu C3   -21.3 % C4    +1.6 % | T1 mu C3    +8.8 % C4   -37.0 % | T2 beta C3   -37.7 % C4   -20.4 %
    S008  T2 mu C3   -41.1 % C4   -30.2 % | T1 mu C3   -34.6 % C4   -41.5 % | T2 beta C3   -12.1 % C4   -18.2 %
    S009  T2 mu C3    +6.6 % C4   -12.3 % | T1 mu C3   +13.5 % C4    -8.5 % | T2 beta C3   +11.7 % C4    +9.2 %
    S010  T2 mu C3   -24.9 % C4    -9.4 % | T1 mu C3   -22.3 % C4   -42.2 % | T2 beta C3   -21.7 % C4   -20.5 %
    cohort median, right-hand imagery: C3 -23.1 % (range -47.9 to +6.6), C4 -11.4 % (range -30.2 to +1.6)
    NOTE: nb-4-4-erd quotes S001 alone at C3 -47.9 % / C4 -10.5 %. S001 was CHOSEN for the size of its effect, and the cohort median above is where a typical subject sits. Quoting the one without the other is the selection effect Level 6 is about.

DELIVERABLE 2 -- contralateral lateralisation: 6 of 10 subjects show it for right-hand imagery (binomial p = 0.7539) and 5 of 10 for left-hand imagery (p = 1.0000).

DELIVERABLE 3 -- cluster test on the difference:
    55 candidate clusters, 0 at p < .05, 1024 permutations (exact for 10 subjects), cluster-forming |t| > 2.262.
        cluster 5: p = 0.0859, 72 cells, 8-15 Hz, +0.47..+0.80 s
    A-priori ROI (one test, chosen before the data): t(9) = +1.858, p = 0.0961, mean +15.257 percentage points, dz +0.588.
    INTERPRETATION: a significant cluster rejects the null of no difference ANYWHERE in the tested window. It does not locate the effect in time or frequency, and the cluster's extent would move with the cluster-forming threshold (nb-4-7-tf-cluster shows by how much).

DELIVERABLE 4 -- who has a mu peak:
    S001  C3 peak  12.71 Hz (exponent 1.26, r^2 0.993) | C4 peak  12.40 Hz (exponent 1.33, r^2 0.992)
    S002  C3 peak  11.85 Hz (exponent 0.79, r^2 0.857) | C4 peak  11.84 Hz (exponent 0.88, r^2 0.988)
    S003  C3 peak   7.45 Hz (exponent 1.32, r^2 0.944) | C4 peak   none Hz (exponent 1.30, r^2 0.986)
    S004  C3 peak   none Hz (exponent 0.55, r^2 0.958) | C4 peak   none Hz (exponent 0.56, r^2 0.958)   <-- NO MU PEAK AT EITHER SITE
    S005  C3 peak   none Hz (exponent 0.90, r^2 0.983) | C4 peak   none Hz (exponent 0.98, r^2 0.983)   <-- NO MU PEAK AT EITHER SITE
    S006  C3 peak   none Hz (exponent 0.28, r^2 0.915) | C4 peak   8.89 Hz (exponent 0.74, r^2 0.914)
    S007  C3 peak  12.36 Hz (exponent 1.19, r^2 0.994) | C4 peak  12.54 Hz (exponent 1.30, r^2 0.991)
    S008  C3 peak  10.41 Hz (exponent 1.09, r^2 0.957) | C4 peak  10.31 Hz (exponent 1.17, r^2 0.972)
    S009  C3 peak   none Hz (exponent 0.51, r^2 0.911) | C4 peak   none Hz (exponent 0.46, r^2 0.583)   <-- NO MU PEAK AT EITHER SITE
    S010  C3 peak  11.77 Hz (exponent 1.61, r^2 0.988) | C4 peak  11.35 Hz (exponent 1.71, r^2 0.990)
    3 of 10 subjects have no fitted peak in 7-13 Hz at either site: S004, S005, S009

RUBRIC:
  [x] Baseline choice justified and edge-safe
      -1.5..-0.5 s, fixed in helpers_l4.TF_BASELINE before the data; the edge region ends at -1.903 s, so the baseline starts 0.403 s clear of it and ends 0.5 s before the cue. Every map shades the edge region.
  [x] Normalisation stated
      percent change from the baseline mean, per frequency, in the BAND-FIRST order (raw power averaged over trials, then over the band, then normalised). nb-4-4-erd shows the three other orders give different answers, so the order is part of the method and is stated wherever a number is.
  [x] Cluster interpretation correct
      the test rejects the null of no difference anywhere in 4-40 Hz x -1.90..+3.90 s; no claim is made about the effect's onset, offset or frequency, and the a-priori ROI is reported beside it as the statement that can be made about a named band and window.
  [x] Subjects without a mu peak handled explicitly
      specparam is fitted on each subject's PRE-CUE data at C3 and C4; 3 of 10 subjects have no peak in 7-13 Hz (S004, S005, S009). They are reported, not dropped; their band-power change is still a change in band power, but it is not evidence about a rhythm.
  [x] Documented defective subjects excluded with the reason recorded
      S088, S089, S092, S100 always and S038, S104 by default, refused by helpers.load_spine with the data/directory.yaml caveat as the reason; none is in this subset.
  [x] Runs on a documented subset within the section 11 limit, with a FULL_COHORT switch
      FULL_COHORT = False; 10 subjects in 835 s here. FULL_COHORT = True is about 103 subjects and ~740 MB and has NOT been executed.

Pitfalls: pf-tf-edge-effects, pf-band-power-slope, pf-cluster-inference-misread, pf-uncorrected-timepoint-tests. Widgets: w-tf-baseline-explorer, w-burst-detector, w-cluster-permutation-viz (mode tf), w-aperiodic-explorer.
TODO(confirm): the catalog carries no published ERD value or lateralisation effect size for this dataset, so nothing above is compared with a literature number; and this dataset's reference is recorded as TODO(confirm) in data/directory.yaml, so the average reference used here is a choice of this course, stated rather than inherited.
deleted 0 downloaded file(s), 0.0 MiB freed
free disk after nb-c4: 4,264 MB (-114 MB against the start of the notebook; the volume is shared, so anything else running on it moves this number too)