Signal quality you can act on live: the gate's criteria cited, its sensitivity on planted blinks, pops, a flat channel and a mains line, three states and a hold, how often a second is unusable on the occipitotemporal replica, the online gate against the offline artifact timeline on the same minute, one consumer export's edge, hole and quality columns, and ICA at rank four measured

nb-7-12-quality Level 7 · Applied Electives ~3 min Used in L7.12 · Signal quality you can act on live

Downloads from ds-eegbci, ds-areeg when you run it.

Download the notebook (.ipynb) Outputs below are the ones stored when it was executed — you do not need to run anything to read it.

nb-7-12-quality · Signal quality you can act on live (L7.12)

Lesson L7.12 · Level 7 · Status draft — for expert review; uncertain points carry TODO(confirm).

Offline you inspect what a cleaner removed and run again. Online you get one pass, so the honest move is to detect a bad stretch and suspend the feedback rather than feed the participant a cleaned estimate of something that was not measurable. This notebook builds that gate, tests it against planted answers, runs it on a real four-channel replica, scores it against the site's offline artifact detector on the same recording, and reads one consumer export to see what its own quality columns are — and are not.

  1. The gate's criteria, cited. Every threshold is ported from data/scripts/detectors.py, the detectors that produced this site's artifact annotations, and the gate prints where each came from.
  2. Planted answers. Blinks, pops, a flat channel and a mains line on the library's synthetic stream, with a truth dict, so sensitivity is a number rather than an impression. The blink sensitivity is a keyed number.
  3. Three states and a hold. Good, suspect, unusable — what each does to the display, and why a hold after a violation is part of the policy.
  4. The occipitotemporal replica, twelve ds-eegbci subjects: how often a one-second window is unusable on four gel channels with no hardware filters. The median fraction is a keyed number.
  5. Online against offline, on the same recording: helpers.detect_artifacts on all 64 channels (its blink rule needs Fp1/Fp2) against the four-channel gate — and what the gate cannot see when the montage has no frontal site.
  6. One consumer export: the ~50 Hz edge and the mains line in its spectrum, and its EQ.* quality columns, shown and never used as truth.
  7. Why ICA is not an option at rank four.

Every number comes from replayed or synthetic data; no headset produced anything here.

In [1]:
# Setup: dependencies, the shared helpers, the course library, non-interactive plotting, a quiet downloader.
import importlib.util
import io
import logging
import re
import subprocess
import sys
import time
import warnings
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path

_needed = ("mne", "scipy", "matplotlib", "pandas", "pooch")
_missing = [p for p in _needed if importlib.util.find_spec(p) is None]
if _missing:
    _req = next((d / "requirements.txt" for d in (Path.cwd(), *Path.cwd().parents)
                 if (d / "requirements.txt").exists()), None)
    _cmd = [sys.executable, "-m", "pip", "install", "-q"]
    _cmd += ["-r", str(_req)] if _req else ["mne==1.10.2", "pooch>=1.8"]
    subprocess.check_call(_cmd)

_shared = next((d / "_shared" for d in (Path.cwd(), *Path.cwd().parents)
                if (d / "_shared" / "helpers.py").exists()), None)
if _shared is None:
    # No checkout around this notebook -- a downloaded .ipynb, or a fresh Colab kernel. Fetch the shared
    # helpers the site publishes beside it. With a checkout present the walk-up above wins and nothing
    # is downloaded; the modules land in ./_shared/ and are reused on every later run.
    import json as _json
    import urllib.request as _urllib
    _base = "https://eeg.neurokinetikz.com/notebooks/_shared/"
    _shared = Path.cwd() / "_shared"
    _shared.mkdir(exist_ok=True)
    for _name in _json.loads(_urllib.urlopen(_base + "index.json").read())["modules"]:
        if not (_shared / _name).exists():
            _urllib.urlretrieve(_base + _name, _shared / _name)
sys.path.insert(0, str(_shared))
import helpers_l7 as L7
_loop = L7.import_eegloop()   # loop/eegloop on sys.path from any directory of a checkout; returns loop/

import helpers
import helpers_l4 as L4
import eegloop
from eegloop.check import bandwidth_edge_hz, line_ratio, run_check
from eegloop.sources import ReplaySource, SyntheticSource, blocks
from eegloop.steps.quality import QUALITY_LABELS, QualityGate

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import mne
from scipy import signal as sps

mne.set_log_level("WARNING")
logging.getLogger("pooch").setLevel(logging.WARNING)
warnings.filterwarnings("ignore", category=RuntimeWarning)
plt.rcParams["figure.dpi"] = 72

SOURCE = "replay"
LOCAL_RECORDING = None      # a path to a recording of your own: a second pass on it, never shipped

DATASET = "ds-eegbci"
OC4 = ["O1", "O2", "T7", "T8"]                 # occipitotemporal-4: exact on the cap
FR4 = ["AF7", "AF8", "TP7", "TP8"]             # the frontal pair of frontotemporal-4 is on the cap; TP7/TP8 stand in for TP9/TP10 (stated)
SUBJECTS = list(range(1, 13))
B = 32
_edf_before = set(Path(mne.get_config("MNE_DATA", str(Path.home() / "mne_data"))).rglob("*.edf"))
print(f"eegloop {eegloop.__version__} · mne {mne.__version__}")
eegloop 0.1.0 · mne 1.10.2

1 · The gate's criteria, cited

The gate runs first, on the raw block — before any band-pass — because every one of its criteria needs what a band-pass removes: the mains line, the 25–100 Hz muscle band, the DC level of a flat channel. It reads a one-second window, decides, and holds the decision for half a second after a violation. Print what it does and where each threshold came from.

In [2]:
gate = QualityGate(256.0, ["TP9", "AF7", "AF8", "TP10"], window_s=1.0, hold_s=0.5, mains_hz=60.0)
d = gate.describe()
print("labels:", QUALITY_LABELS)
print("thresholds:", d["thresholds"])
print("criteria from:", d["criteria_from"])
print("blink detection:", d["blink_detection"], "| pop detection:", d["pop_detection"])
print("verdict:", d["verdict"])
print("latency:", gate.latency_note)


def gate_timeline(x, fs, names, *, mains=None, window_s=1.0, hold_s=0.5, frontal=None):
    # Run the gate over an array as a stream; one row per block with the verdict and the per-channel labels.
    src = ReplaySource(np.asarray(x, float), fs=fs, ch_names=names, pace="fast", block_samples=B)
    kw = {} if frontal is None else {"frontal": frontal}
    g = QualityGate(fs, names, window_s=window_s, hold_s=hold_s, mains_hz=mains, **kw)
    rows = []
    src.start()
    for b in blocks(src, B):
        q = g.process(b).flags["quality"]
        rows.append({"t0": b.t_start_s, "t1": b.t_end_s, "ready": q["ready"], "ok": q["ok"], "state": q["state"],
                     "labels": tuple(q["labels"]), **{f"lab_{c}": tuple(q["per_channel"][c]["labels"]) for c in names}})
    src.stop()
    return pd.DataFrame(rows)
labels: ('flat', 'pop', 'blink', 'emg', 'line-noise', 'gap', 'warming-up')
thresholds: {'flat_sd_uv': 0.5, 'pop_k': 6.0, 'pop_min_step_uv': 100.0, 'pop_jump_uv': 100.0, 'blink_k': 4.0, 'blink_min_amp_uv': 50.0, 'emg_k': 3.0, 'emg_min_rms_uv': 4.0, 'line_min_ratio': 10.0}
criteria from: ['data/scripts/detectors.py', 'notebooks/_shared/helpers.py:detect_artifacts (pop jump)']
blink detection: True | pop detection: True
verdict: ok unless a channel is unusable, fewer than min_good_channels are good-or-suspect, the hold is running, or the window is not yet full; suspect channels are fed back and labelled
latency: decision delay: a 1-s window (plus a 0.5-s hold after a violation); the gate delays the decision, not the signal

2 · Planted answers: how sensitive is the gate?

The synthetic stream plants blinks on the frontal pair, single-channel pops, a flat channel and a mains line, and records every one in truth. A blink is found when a block within 1.5 s of its onset carries the label; a false alarm is a labelled run outside every planted window. The blink scenario is run with 27 blinks rather than the default four so the fraction means something.

In [3]:
def runs_of(df, label, channel=None):
    col = "labels" if channel is None else f"lab_{channel}"
    hit = df[col].apply(lambda ls: label in ls).to_numpy()
    out, start = [], None
    for i, h in enumerate(hit):
        if h and start is None:
            start = i
        if not h and start is not None:
            out.append((df.t0.iloc[start], df.t1.iloc[i - 1])); start = None
    if start is not None:
        out.append((df.t0.iloc[start], df.t1.iloc[len(df) - 1]))
    return out

def sensitivity(df, label, onsets, *, before=0.2, after=1.5):
    found = [any((r0 <= o + after) and (r1 >= o - before) for r0, r1 in runs_of(df, label)) for o in onsets]
    inside = lambda r: any((r[0] <= o + after) and (r[1] >= o - before) for o in onsets)
    false = [r for r in runs_of(df, label) if not inside(r)]
    return sum(found), len(onsets), len(false)

blink_times = tuple(float(t) for t in np.arange(5.0, 175.0, 6.5))
syn_blinks = SyntheticSource("blinks", duration_s=180.0, blink_times_s=blink_times, pace="fast")
tl_blinks = gate_timeline(syn_blinks._data, syn_blinks.info.fs, list(syn_blinks.info.ch_names), mains=syn_blinks.truth["mains_hz"])
n_found, n_planted, n_false = sensitivity(tl_blinks, "blink", syn_blinks.truth["blink_onsets_s"])
blink_sens = n_found / n_planted
print(f"blinks:  {n_found}/{n_planted} planted blinks labelled within 1.5 s of onset (sensitivity {blink_sens:.2f}); {n_false} blink run(s) outside any planted window")

syn_pops = SyntheticSource("pops", duration_s=90.0, pace="fast")
tl_pops = gate_timeline(syn_pops._data, syn_pops.info.fs, list(syn_pops.info.ch_names), mains=syn_pops.truth["mains_hz"])
pop_found, pop_n, pop_false = sensitivity(tl_pops, "pop", [p["onset_s"] for p in syn_pops.truth["pops"]], after=2.0)
print(f"pops:    {pop_found}/{pop_n} planted pops labelled (on channels {[p['channel'] for p in syn_pops.truth['pops']]}); {pop_false} pop run(s) elsewhere")

syn_flat = SyntheticSource("flat-channel", duration_s=60.0, pace="fast")
tl_flat = gate_timeline(syn_flat._data, syn_flat.info.fs, list(syn_flat.info.ch_names), mains=syn_flat.truth["mains_hz"])
ready = tl_flat[tl_flat.ready]
flat_frac = ready[f"lab_{syn_flat.truth['flat_channel']}"].apply(lambda ls: "flat" in ls).mean()
others_flat = max(ready[f"lab_{c}"].apply(lambda ls: "flat" in ls).mean() for c in syn_flat.info.ch_names if c != syn_flat.truth["flat_channel"])
print(f"flat:    {syn_flat.truth['flat_channel']} labelled flat in {flat_frac * 100:.0f} % of decisions; the other channels at most {others_flat * 100:.0f} %")

syn_line = SyntheticSource("line-noise", duration_s=60.0, pace="fast")
tl_line = gate_timeline(syn_line._data, syn_line.info.fs, list(syn_line.info.ch_names), mains=syn_line.truth["mains_hz"])
ready = tl_line[tl_line.ready]
print(f"mains:   line-noise (a SUSPECT label: fed back, and said so) on {ready.labels.apply(lambda ls: 'line-noise' in ls).mean() * 100:.0f} % of decisions; "
      f"state unusable on {(ready.state == 'unusable').mean() * 100:.0f} % (the gate does not close on a mains line)")
blinks:  27/27 planted blinks labelled within 1.5 s of onset (sensitivity 1.00); 0 blink run(s) outside any planted window
pops:    2/2 planted pops labelled (on channels ['TP9', 'AF8']); 0 pop run(s) elsewhere
flat:    TP10 labelled flat in 100 % of decisions; the other channels at most 0 %
mains:   line-noise (a SUSPECT label: fed back, and said so) on 100 % of decisions; state unusable on 0 % (the gate does not close on a mains line)
In [4]:
fig, ax = plt.subplots(figsize=(13, 3.6))
t = np.arange(syn_blinks._data.shape[1]) / syn_blinks.info.fs
ax.plot(t, syn_blinks._data[1], lw=0.6, color="0.3", label="AF7 (µV)")
for o in syn_blinks.truth["blink_onsets_s"]:
    ax.axvline(o, color="C1", lw=0.8, alpha=0.6)
for r0, r1 in runs_of(tl_blinks, "blink"):
    ax.axvspan(r0, r1, color="C0", alpha=0.25, lw=0)
ax.set(xlim=(0, 90), xlabel="Time (s)", ylabel="Amplitude (µV)",
       title=f"Planted blink onsets (orange) and the gate's blink windows (blue): {n_found}/{n_planted} found, {n_false} false runs")
ax.legend(loc="upper right"); plt.tight_layout(); plt.show()
Figure 1 of notebook nb-7-12-quality, an output plot. The text around it states what it shows and the units of every axis.

3 · Three states and a hold

The verdict has three states. Unusable — a flat, popped, blinking, muscle-laden channel, or a block after a loss — freezes or zeroes the display, and the gate holds that verdict for half a second after the last violation so the display does not flicker at the edge of an artifact. Suspect — a visible mains line — is fed back and labelled, because a mains line is the normal state of an un-notched recording and the band-pass removes it; the label travels with the session log. Good is the rest. Watch the state around one blink.

In [5]:
o = syn_blinks.truth["blink_onsets_s"][3]
win = tl_blinks[(tl_blinks.t0 >= o - 1.5) & (tl_blinks.t1 <= o + 2.5)]
print(f"around the blink at {o:.1f} s (blocks of {B} samples = {B / syn_blinks.info.fs * 1000:.0f} ms):")
for _, r in win.iterrows():
    print(f"  {r.t0:6.2f}–{r.t1:6.2f} s  {r.state:9s} ok={str(r.ok):5s} labels={list(r.labels)}")
n_unusable = int((win.state == "unusable").sum())
print(f"\n{n_unusable} blocks unusable = {n_unusable * B / syn_blinks.info.fs:.2f} s: the blink itself, and the {gate.hold_s:g}-s hold after its last violation")
around the blink at 24.5 s (blocks of 32 samples = 125 ms):
   23.00– 23.12 s  suspect   ok=True  labels=['line-noise']
   23.12– 23.25 s  suspect   ok=True  labels=['line-noise']
   23.25– 23.38 s  suspect   ok=True  labels=['line-noise']
   23.38– 23.50 s  suspect   ok=True  labels=['line-noise']
   23.50– 23.62 s  suspect   ok=True  labels=['line-noise']
   23.62– 23.75 s  suspect   ok=True  labels=['line-noise']
   23.75– 23.88 s  suspect   ok=True  labels=['line-noise']
   23.88– 24.00 s  suspect   ok=True  labels=['line-noise']
   24.00– 24.12 s  suspect   ok=True  labels=['line-noise']
   24.12– 24.25 s  suspect   ok=True  labels=['line-noise']
   24.25– 24.38 s  suspect   ok=True  labels=['line-noise']
   24.38– 24.50 s  unusable  ok=False labels=['blink', 'line-noise']
   24.50– 24.62 s  unusable  ok=False labels=['blink', 'line-noise']
   24.62– 24.75 s  unusable  ok=False labels=['blink', 'line-noise']
   24.75– 24.88 s  unusable  ok=False labels=['line-noise']
   24.88– 25.00 s  unusable  ok=False labels=['line-noise']
   25.00– 25.12 s  unusable  ok=False labels=['line-noise']
   25.12– 25.25 s  suspect   ok=True  labels=['line-noise']
   25.25– 25.38 s  suspect   ok=True  labels=['line-noise']
   25.38– 25.50 s  suspect   ok=True  labels=['line-noise']
   25.50– 25.62 s  suspect   ok=True  labels=['line-noise']
   25.62– 25.75 s  suspect   ok=True  labels=['line-noise']
   25.75– 25.88 s  suspect   ok=True  labels=['line-noise']
   25.88– 26.00 s  suspect   ok=True  labels=['line-noise']
   26.00– 26.12 s  suspect   ok=True  labels=['line-noise']
   26.12– 26.25 s  suspect   ok=True  labels=['line-noise']
   26.25– 26.38 s  suspect   ok=True  labels=['line-noise']
   26.38– 26.50 s  suspect   ok=True  labels=['line-noise']
   26.50– 26.62 s  suspect   ok=True  labels=['line-noise']
   26.62– 26.75 s  suspect   ok=True  labels=['line-noise']
   26.75– 26.88 s  suspect   ok=True  labels=['line-noise']
   26.88– 27.00 s  suspect   ok=True  labels=['line-noise']

6 blocks unusable = 0.75 s: the blink itself, and the 0.5-s hold after its last violation

4 · The occipitotemporal replica: how often is a second unusable?

ds-eegbci R01 (eyes open, one minute) on O1, O2, T7, T8 for twelve subjects — gel electrodes, a research amplifier, no hardware filters. No frontal site, so no blink label can occur; what closes the gate here is muscle, a pop, or a flat channel. The median fraction of one-second decisions called unusable is a keyed number. It is a fact about this replica; on a dry headband at the same sites the number is measured the same way and belongs to that recording's log.

In [6]:
t0 = time.time()
rows = []
for s in SUBJECTS:
    raw = helpers.load_spine(DATASET, s, run=1)
    x = raw.get_data(picks=OC4) * 1e6
    x = x - x.mean(axis=1, keepdims=True)
    tl = gate_timeline(x, float(raw.info["sfreq"]), OC4, mains=60.0)
    ready = tl[tl.ready]
    labs = {lab: float(ready.labels.apply(lambda ls: lab in ls).mean()) for lab in QUALITY_LABELS if lab != "warming-up"}
    rows.append({"subject": f"S{s:03d}", "decisions": len(ready), "unusable": float((ready.state == "unusable").mean()),
                 "suspect": float((ready.state == "suspect").mean()), **{k: v for k, v in labs.items() if v > 0}})
rep = pd.DataFrame(rows).fillna(0.0)
unusable_median = float(rep.unusable.median())
print(f"{len(rep)} subjects, R01 eyes open, occipitotemporal-4, gate window 1 s / hold 0.5 s, mains 60 Hz  ({time.time() - t0:.1f} s)")
print(f"fraction of decisions UNUSABLE: median {unusable_median:.3f}, range {rep.unusable.min():.3f}–{rep.unusable.max():.3f}")
print(f"fraction SUSPECT (line-noise, fed back): median {rep.suspect.median():.3f}")
print(rep.round(3).to_string(index=False))
worst = rep.sort_values("unusable").iloc[-1]
print(f"\nthe subject with the highest fraction is {worst.subject} at {worst.unusable:.3f}: " +
      ", ".join(f"{k} {worst[k]:.2f}" for k in ("flat", "pop", "emg", "blink") if k in rep.columns and worst[k] > 0) +
      " — a replica channel that is unusable for the whole minute is a fact about that recording, and the gate says so rather than averaging it away")
12 subjects, R01 eyes open, occipitotemporal-4, gate window 1 s / hold 0.5 s, mains 60 Hz  (2.0 s)
fraction of decisions UNUSABLE: median 0.053, range 0.003–1.000
fraction SUSPECT (line-noise, fed back): median 0.468
subject  decisions  unusable  suspect  flat   pop  line-noise
   S001        301     0.090    0.339 0.007 0.030       0.362
   S002        301     1.000    0.000 0.007 0.920       0.123
   S003        301     0.153    0.100 0.007 0.050       0.110
   S004        301     0.007    0.993 0.007 0.000       0.997
   S005        301     0.007    0.993 0.007 0.000       0.997
   S006        301     0.003    0.997 0.003 0.000       1.000
   S007        301     0.206    0.794 0.007 0.070       0.997
   S008        301     0.007    0.605 0.007 0.000       0.605
   S009        301     0.993    0.007 0.007 0.734       0.997
   S010        301     0.462    0.266 0.007 0.189       0.385
   S011        301     0.017    0.236 0.007 0.003       0.236
   S012        301     0.007    0.598 0.007 0.000       0.598

the subject with the highest fraction is S002 at 1.000: flat 0.01, pop 0.92 — a replica channel that is unusable for the whole minute is a fact about that recording, and the gate says so rather than averaging it away

5 · Online against offline, on the same recording

The site's offline detector (helpers.detect_artifacts) sees all 64 channels of S001 R01 and labels blinks from Fp1 and Fp2, the sites where a blink is largest. The online gate sees four. Run both on the same minute, twice: once on the occipitotemporal replica, where no frontal site exists and the gate says so, and once on a four-channel set whose frontal pair is the exact AF7/AF8 of the frontotemporal geometry — the cap has those two — with TP7/TP8 standing in for the TP9/TP10 it lacks (a stated substitution, used for nothing but the gate's non-frontal channels). Then score the online blink windows against the offline blink events.

In [7]:
t0 = time.time()
raw_full = helpers.load_spine(DATASET, 1, run=1)
fs = float(raw_full.info["sfreq"])
offline = helpers.detect_artifacts(raw_full, mains_hz=60.0)
off_blinks = [e for e in offline if e["label"] == "blink"]
print(f"offline reference on 64 channels: {len(offline)} events; {len(off_blinks)} blinks from Fp1/Fp2  ({time.time() - t0:.1f} s)")
print("  labels:", {lab: sum(e['label'] == lab for e in offline) for lab in helpers.ARTIFACT_LABELS if any(e['label'] == lab for e in offline)})

def subset_uv(raw, chs):
    x = raw.get_data(picks=chs) * 1e6
    return x - x.mean(axis=1, keepdims=True)

tl_oc = gate_timeline(subset_uv(raw_full, OC4), fs, OC4, mains=60.0)
tl_fr = gate_timeline(subset_uv(raw_full, FR4), fs, FR4, mains=60.0, frontal=("AF7", "AF8"))
g_oc = QualityGate(fs, OC4, mains_hz=60.0); g_fr = QualityGate(fs, FR4, mains_hz=60.0, frontal=("AF7", "AF8"))
print(f"\nocc. replica gate says: blink detection = {g_oc.describe()['blink_detection']!r}")
print(f"frontal-pair gate says: blink detection = {g_fr.describe()['blink_detection']!r}")

on_blinks_fr = runs_of(tl_fr, "blink")
on_blinks_oc = runs_of(tl_oc, "blink")
def match(events, runs, slack=0.5):
    hit = [any(r0 <= e["onset_s"] + e["duration_s"] + slack and r1 >= e["onset_s"] - slack for r0, r1 in runs) for e in events]
    extra = [r for r in runs if not any(r[0] <= e["onset_s"] + e["duration_s"] + slack and r[1] >= e["onset_s"] - slack for e in events)]
    return sum(hit), len(events), len(extra)
h, n, extra = match(off_blinks, on_blinks_fr)
print(f"\nfrontal-pair gate (AF7/AF8): {h}/{n} offline blinks matched by an online blink window; {len(on_blinks_fr)} online windows, {extra} matching no offline event")
print(f"occipitotemporal gate:       {len(on_blinks_oc)} online blink windows (none are possible: no frontal site)")
print("The two detectors use different sites (Fp1/Fp2 offline, AF7/AF8 online), different windows (0.4-s events against 1-s decisions with a hold) and the same")
print("amplitude criterion in spirit; a mismatch here is a fact about what four channels at those sites can see, which is L7.5's point.")
offline reference on 64 channels: 97 events; 18 blinks from Fp1/Fp2  (0.1 s)
  labels: {'blink': 18, 'saccade': 45, 'drift': 29, 'pop': 2, 'line-noise': 3}
occ. replica gate says: blink detection = 'no frontal channel in this montage'
frontal-pair gate says: blink detection = True

frontal-pair gate (AF7/AF8): 11/18 offline blinks matched by an online blink window; 10 online windows, 1 matching no offline event
occipitotemporal gate:       0 online blink windows (none are possible: no frontal site)
The two detectors use different sites (Fp1/Fp2 offline, AF7/AF8 online), different windows (0.4-s events against 1-s decisions with a hold) and the same
amplitude criterion in spirit; a mismatch here is a fact about what four channels at those sites can see, which is L7.5's point.
In [8]:
T = raw_full.n_times / fs
fig, axes = plt.subplots(2, 1, figsize=(13, 5.2), sharex=True, gridspec_kw={"height_ratios": [1.6, 1]})
helpers.artifact_timeline(offline, T, title="Offline reference: helpers.detect_artifacts on all 64 channels (S001 R01)", ax=axes[0])
ax = axes[1]
for i, (name, tl) in enumerate((("frontal pair (AF7/AF8 + TP7/TP8)", tl_fr), ("occipitotemporal-4", tl_oc))):
    for lab, c in (("blink", "C0"), ("emg", "C3"), ("pop", "C2"), ("flat", "C4")):
        for r0, r1 in runs_of(tl, lab):
            ax.barh(i, r1 - r0, left=r0, height=0.6, color=c, lw=0)
    ax.text(0.5, i + 0.42, name, fontsize=9, va="bottom")
ax.set(yticks=[0, 1], yticklabels=["frontal pair", "occ.-temp."], ylim=(-0.5, 1.9), xlabel="Time (s)", xlim=(0, T),
       title="Online gate, unusable windows by label (blink blue, emg red, pop green, flat purple)")
plt.tight_layout(); plt.show()
Figure 2 of notebook nb-7-12-quality, an output plot. The text around it states what it shows and the units of every axis.

6 · One consumer export: the edge, the mains line, and the columns that are not signal

ds-areeg (CC BY 4.0) is the directory's only downloadable consumer front-end: fourteen saline channels at 128 Hz, exported by the vendor's software as a CSV whose 67 columns include the 14 EEG channels beside contact-quality (CQ.*), signal-quality (EQ.*) and motion (MOT.*) families. One participant's first recordings are fetched by HTTP range request from the archive's central directory — about a hundred kilobytes each of a 32 MB zip — and run through the same acceptance report as every other stream on this site. The report's bandwidth edge and line ratios say what the front-end did before the file existed; the EQ.* columns are shown beside an 8–12 Hz power the notebook computes from the raw channels, and are used as truth for nothing.

In [9]:
AREEG_CHANNELS, AREEG_SFREQ, AREEG_ZIP_BYTES = L7.AREEG_CHANNELS, L7.AREEG_SFREQ, L7.AREEG_ZIP_BYTES   # the shared loader (helpers_l7, Phase 5)
N_RECORDINGS = 3

t0 = time.time()
areeg_ix = L7.areeg_index()
members = L7.areeg_participant(areeg_ix, 1, n=N_RECORDINGS)

with ThreadPoolExecutor(max_workers=3) as ex:
    loaded = list(ex.map(lambda e: L7.areeg_load(areeg_ix, e), members))
print(f"ds-areeg par.1: {len(loaded)} recordings fetched by range request ({sum(e['compressed_bytes'] for e in members) / 1e3:.0f} kB of {AREEG_ZIP_BYTES / 1e6:.1f} MB), {time.time() - t0:.0f} s")
meta0, df0 = loaded[0]
print("header (selected keys; the export also carries the device's serial, which is not printed):",
      {k: meta0[k] for k in ("title", "start timestamp", "headset type") if k in meta0})
x_ar = np.concatenate([L7.areeg_eeg(df) for _, df in loaded], axis=1)
print(f"EEG: {x_ar.shape[0]} channels × {x_ar.shape[1]} samples at {AREEG_SFREQ:g} Hz = {x_ar.shape[1] / AREEG_SFREQ:.0f} s (three recordings concatenated for the report; each is a separate export)")

src_ar = ReplaySource(x_ar, fs=AREEG_SFREQ, ch_names=list(AREEG_CHANNELS), pace="fast", block_samples=B, kind="replay",
                      nominal={"dataset": "ds-areeg", "device_class": "consumer", "hardware_filters": "TODO(confirm): the vendor front-end; not documented in the paper", "expected_fs": AREEG_SFREQ})
report_ar = run_check(src_ar, seconds=x_ar.shape[1] / AREEG_SFREQ - 1, mains_hz=None)
for r in report_ar["rows"]:
    if r["item"] in ("sampling rate", "flat channels", "line noise", "bandwidth edge", "quality gate"):
        print(f"[{r['status']:4s}] {r['item']:16s} {r['value'][:150]}")
edges = [r["bandwidth_edge_hz"] for r in report_ar["per_channel"]]
print(f"\nbandwidth edge per channel: {[None if e is None else round(e) for e in edges]} Hz  (Nyquist {AREEG_SFREQ / 2:g} Hz)")
print(f"line ratio at 50 Hz: median {np.median([r['line_50_ratio'] for r in report_ar['per_channel']]):.1f}; at 60 Hz: median {np.median([r['line_60_ratio'] for r in report_ar['per_channel']]):.1f}")
ds-areeg par.1: 3 recordings fetched by range request (309 kB of 32.2 MB), 6 s
header (selected keys; the export also carries the device's serial, which is not printed): {'title': 'aya اختر', 'start timestamp': '1702803254.654390', 'headset type': 'EPOCX'}
EEG: 14 channels × 4799 samples at 128 Hz = 37 s (three recordings concatenated for the report; each is a separate export)
[info] sampling rate    128 Hz claimed
[ok  ] flat channels    none
[warn] line noise       strongest at 50 Hz on P8: ratio 99.4
[info] bandwidth edge   the spectrum falls 20 dB below its 4–20 Hz level from 50 Hz on
[info] quality gate     143 decisions; labels per channel: AF3 blink 1 %; F7 none; F3 none; FC5 emg 16 %; T7 none; P7 none; O1 none; O2 none; P8 none; T8 none; FC6 none; F4 n

bandwidth edge per channel: [51, 51, 51, 51, 51, 51, 50, 51, 51, 51, 51, 51, 51, 51] Hz  (Nyquist 64 Hz)
line ratio at 50 Hz: median 46.6; at 60 Hz: median 0.5
In [10]:
# The spectrum, the edge, the mains line — and the vendor's quality column beside a stated computation.
fig, axes = plt.subplots(1, 2, figsize=(13.5, 4.2))
ax = axes[0]
for i, c in enumerate(AREEG_CHANNELS):
    f, p = sps.welch(x_ar[i], fs=AREEG_SFREQ, nperseg=int(2 * AREEG_SFREQ))
    ax.semilogy(f, p, lw=0.8, color="C0" if c in ("O1", "O2") else "0.6", alpha=1 if c in ("O1", "O2") else 0.5)
edge_o1 = report_ar["per_channel"][AREEG_CHANNELS.index("O1")]["bandwidth_edge_hz"]
if edge_o1:
    ax.axvline(edge_o1, color="C1", ls="--", lw=1, label=f"bandwidth edge on O1: {edge_o1:.0f} Hz")
ax.axvline(50, color="0.4", ls=":", lw=0.8); ax.axvline(60, color="0.4", ls=":", lw=0.8)
ax.set(xlabel="Frequency (Hz)", ylabel="Power (µV²/Hz)", xlim=(1, AREEG_SFREQ / 2), title="ds-areeg par.1 (14 ch, 128 Hz): O1/O2 in blue")
ax.legend(loc="lower left", fontsize=8)

ax = axes[1]
df = df0
o1 = df["EEG.O1"].to_numpy(float); o1 = o1 - o1.mean()
sec = np.arange(0, o1.size // int(AREEG_SFREQ)) * 1.0
alpha_pow = []
for k in range(o1.size // int(AREEG_SFREQ)):
    seg = o1[k * int(AREEG_SFREQ):(k + 1) * int(AREEG_SFREQ)]
    f, p = sps.welch(seg, fs=AREEG_SFREQ, nperseg=int(AREEG_SFREQ))
    alpha_pow.append(float(p[(f >= 8) & (f <= 12)].mean()))
ax.plot(sec, alpha_pow, color="C0", lw=1.4, label="8–12 Hz power from EEG.O1 (Welch, 1-s windows) — a stated computation")
ax2 = ax.twinx()
eq = df["EQ.O1"].dropna()
ax2.step(eq.index / AREEG_SFREQ, eq.values, where="post", color="C3", lw=1.4, label="EQ.O1 — the vendor's signal-quality column (its scale)")
ax.set(xlabel="Time in recording (s)", ylabel="8–12 Hz power (µV²/Hz)", title="A number you can state beside one you cannot")
ax2.set_ylabel("EQ.O1 (vendor units)")
h1, l1 = ax.get_legend_handles_labels(); h2, l2 = ax2.get_legend_handles_labels()
ax.legend(h1 + h2, l1 + l2, loc="upper right", fontsize=8)
plt.tight_layout(); plt.show()
cols = df.columns
print(f"{len(cols)} columns in the export; EEG signal columns: {sum(c.startswith('EEG.') and c.split('.', 1)[1] in AREEG_CHANNELS for c in cols)}; "
      f"CQ.*: {sum(c.startswith('CQ.') for c in cols)}; EQ.*: {sum(c.startswith('EQ.') for c in cols)}; MOT.*: {sum(c.startswith('MOT.') for c in cols)}")
print(f"EQ.O1 is written on {100 * df['EQ.O1'].notna().mean():.1f} % of rows (a periodic firmware report), values {sorted(df['EQ.O1'].dropna().unique().tolist())[:8]} …")
Figure 3 of notebook nb-7-12-quality, an output plot. The text around it states what it shows and the units of every axis.
68 columns in the export; EEG signal columns: 14; CQ.*: 15; EQ.*: 16; MOT.*: 12
EQ.O1 is written on 1.6 % of rows (a periodic firmware report), values [1.0, 2.0] …

7 · Why ICA is not an option at rank four

Independent component analysis returns at most as many components as there are channels. On a 64-channel cap a blink has a component of its own and removing it costs the EEG almost nothing (L2.6). On four channels there are four components to hold a blink, a muscle burst, the mains line, the alpha rhythm and everything else at once, so the component that carries the blink carries EEG too, and removing it removes signal. Measure that on the same minute: find the component whose time course follows the offline blink events, remove it, and ask how much of the 8–12 Hz power at the two non-frontal channels survived — at rank four, and at rank 64.

In [11]:
from mne.preprocessing import ICA

def blink_series(n, fs, events):
    y = np.zeros(n)
    for e in events:
        a, b = int(e["onset_s"] * fs), int((e["onset_s"] + e["duration_s"]) * fs)
        y[a:b] = 1.0
    return y

def alpha_power(raw, chs):
    x = raw.get_data(picks=chs) * 1e6
    f, p = sps.welch(x, fs=raw.info["sfreq"], nperseg=int(2 * raw.info["sfreq"]))
    return float(p[:, (f >= 8) & (f <= 12)].mean())

def blink_ptp(raw, ch, events):
    # median peak-to-peak of the 0.5-15 Hz signal at `ch` inside each offline blink event
    x = raw.copy().pick([ch]).filter(0.5, 15.0, verbose=False).get_data()[0] * 1e6
    vals = [np.ptp(x[int(e["onset_s"] * fs):int((e["onset_s"] + e["duration_s"]) * fs)]) for e in events]
    return float(np.median(vals))

blinks_y = blink_series(raw_full.n_times, fs, off_blinks)
rows = []
for label, chs, n_comp in (("rank 4 (AF7, AF8, TP7, TP8)", FR4, 4), ("rank 64 (the whole cap)", None, 20)):
    r = raw_full.copy().filter(1.0, None, verbose=False)
    if chs is not None:
        r.pick(chs)
    ica = ICA(n_components=n_comp, random_state=20260920, max_iter="auto", verbose=False).fit(r)
    src = ica.get_sources(r).get_data()
    corr = [abs(np.corrcoef(src[k], blinks_y)[0, 1]) for k in range(src.shape[0])]
    k = int(np.argmax(corr))
    cleaned = ica.apply(r.copy(), exclude=[k], verbose=False)
    rows.append({"montage": label, "components": n_comp, "blink component": k, "r with offline blinks": round(corr[k], 2),
                 "blink p-p at AF7 removed": round(1 - blink_ptp(cleaned, "AF7", off_blinks) / blink_ptp(r, "AF7", off_blinks), 3),
                 "8-12 Hz at TP7/TP8 retained": round(alpha_power(cleaned, ["TP7", "TP8"]) / alpha_power(r, ["TP7", "TP8"]), 3)})
ica_tab = pd.DataFrame(rows)
print(ica_tab.to_string(index=False))
r4, r64 = ica_tab.iloc[0], ica_tab.iloc[1]
retained_4, retained_64 = float(r4["8-12 Hz at TP7/TP8 retained"]), float(r64["8-12 Hz at TP7/TP8 retained"])
removed_4, removed_64 = float(r4["blink p-p at AF7 removed"]), float(r64["blink p-p at AF7 removed"])
print()
print(f"At rank 64 the component that follows the blinks (r = {r64['r with offline blinks']}) removes {removed_64 * 100:.0f} % of the blink amplitude at AF7 and keeps "
      f"{retained_64 * 100:.0f} % of the temporal alpha. At rank 4 (r = {r4['r with offline blinks']}) it removes {removed_4 * 100:.0f} % and keeps {retained_4 * 100:.0f} %.")
print("What the numbers say on THIS minute is printed above and is not tuned. The structural objection stands whatever they say: four components must hold")
print("every source at once, the component that 'is the blink' is whatever mixture the fit found, and online nobody inspects it. The policy gates rather than cleans.")
                    montage  components  blink component  r with offline blinks  blink p-p at AF7 removed  8-12 Hz at TP7/TP8 retained
rank 4 (AF7, AF8, TP7, TP8)           4                0                   0.39                     0.687                        0.997
    rank 64 (the whole cap)          20                0                   0.39                     0.591                        0.995

At rank 64 the component that follows the blinks (r = 0.39) removes 59 % of the blink amplitude at AF7 and keeps 100 % of the temporal alpha. At rank 4 (r = 0.39) it removes 69 % and keeps 100 %.
What the numbers say on THIS minute is printed above and is not tuned. The structural objection stands whatever they say: four components must hold
every source at once, the component that 'is the blink' is whatever mixture the fit found, and online nobody inspects it. The policy gates rather than cleans.

Your own recording, if you have one

When LOCAL_RECORDING names a file — a .bin with its sidecar as eegloop run --record-raw writes it, or an .npz — the cell below runs this notebook's measurement on it and prints the result titled as yours; when it is None the cell is skipped entirely, so nothing empty ships. Nothing from your recording is written into this notebook's outputs on the site: it is a second pass, never shipped (CONTRACTS, Phase 5).

In [12]:
if LOCAL_RECORDING is not None:
    mine = ReplaySource(LOCAL_RECORDING, pace="fast", block_samples=32)
    gate = QualityGate(mine.info.fs, mine.info.ch_names, window_s=1.0, hold_s=0.5, mains_hz=60.0)
    states, labels = [], {}
    for b in blocks(mine, 32):
        q = gate.process(b).flags["quality"]
        states.append(q["state"])
        for lab in q["labels"]:
            labels[lab] = labels.get(lab, 0) + 1
    n = len(states)
    print(f"your recording — not shipped: {n} blocks; unusable {states.count('unusable') / n:.3f}, suspect {states.count('suspect') / n:.3f}, good {states.count('good') / n:.3f}")
    print("labels seen (blocks):", dict(sorted(labels.items())))
else:
    print("LOCAL_RECORDING is None: skipped (set it to a .bin with its sidecar, or an .npz, to run the gate on your own file)")
LOCAL_RECORDING is None: skipped (set it to a .bin with its sidecar, or an .npz, to run the gate on your own file)

8 · The numbers the lesson's exercises ask for

In [13]:
print("nb-7-12-quality — L7.12 numbers (draft; TODO(confirm) at author review)")
print()
print(f"(a) Fraction of one-second decisions called UNUSABLE on the occipitotemporal-4 replica, ds-eegbci S001–S{len(rep):03d} R01 (eyes open), "
      f"gate window 1 s / hold 0.5 s: median {unusable_median:.3f} (range {rep.unusable.min():.3f}–{rep.unusable.max():.3f}).")
print(f"(b) Online blink sensitivity against planted blinks (synthetic 'blinks' scenario, {n_planted} blinks on AF7/AF8): {n_found}/{n_planted} = {blink_sens:.2f}; "
      f"{n_false} false blink run(s).")
print(f"(c) ICA on the same minute: the component that follows the offline blinks removes {removed_4 * 100:.0f} % of the blink amplitude at AF7 and keeps "
      f"{retained_4 * 100:.0f} % of the 8–12 Hz power at TP7/TP8 at rank 4, against {removed_64 * 100:.0f} % and {retained_64 * 100:.0f} % at rank 64. "
      "The objection to online ICA at rank four is structural (four components for every source) and procedural (nobody inspects the fit); the policy gates rather than cleans.")
print()
print(f"Offline-against-online on S001 R01: {h}/{n} offline blinks (Fp1/Fp2) matched by the frontal-pair gate (AF7/AF8); the occipitotemporal gate cannot label a blink.")
print(f"ds-areeg par.1: bandwidth edge on O1 {edge_o1 if edge_o1 is None else round(edge_o1)} Hz of a {AREEG_SFREQ / 2:g} Hz Nyquist; "
      f"line ratio medians 50 Hz {np.median([r['line_50_ratio'] for r in report_ar['per_channel']]):.1f}, 60 Hz {np.median([r['line_60_ratio'] for r in report_ar['per_channel']]):.1f}.")
nb-7-12-quality — L7.12 numbers (draft; TODO(confirm) at author review)

(a) Fraction of one-second decisions called UNUSABLE on the occipitotemporal-4 replica, ds-eegbci S001–S012 R01 (eyes open), gate window 1 s / hold 0.5 s: median 0.053 (range 0.003–1.000).
(b) Online blink sensitivity against planted blinks (synthetic 'blinks' scenario, 27 blinks on AF7/AF8): 27/27 = 1.00; 0 false blink run(s).
(c) ICA on the same minute: the component that follows the offline blinks removes 69 % of the blink amplitude at AF7 and keeps 100 % of the 8–12 Hz power at TP7/TP8 at rank 4, against 59 % and 100 % at rank 64. The objection to online ICA at rank four is structural (four components for every source) and procedural (nobody inspects the fit); the policy gates rather than cleans.

Offline-against-online on S001 R01: 11/18 offline blinks (Fp1/Fp2) matched by the frontal-pair gate (AF7/AF8); the occipitotemporal gate cannot label a blink.
ds-areeg par.1: bandwidth edge on O1 50 Hz of a 64 Hz Nyquist; line ratio medians 50 Hz 46.6, 60 Hz 0.5.

Cleanup

Only what this run fetched is deleted; the archive members were inflated in memory and never written to disk.

In [14]:
_edf_after = set(Path(mne.get_config("MNE_DATA", str(Path.home() / "mne_data"))).rglob("*.edf"))
_new = sorted(_edf_after - _edf_before)
for p in _new:
    p.unlink(missing_ok=True)
assert len(rep) == len(SUBJECTS), f"cohort incomplete: {len(rep)} of {len(SUBJECTS)}"
print(f"deleted {len(_new)} EDF file(s) this run fetched; kept everything that was already cached; cohort {len(rep)}/{len(SUBJECTS)}")
deleted 0 EDF file(s) this run fetched; kept everything that was already cached; cohort 12/12