nb-7-14-protocol · A protocol that can fail: sham, blinding, pre-registration, the valid test (L7.14)¶
Lesson L7.14 · Level 7 · Status draft — for expert review; uncertain points carry TODO(confirm).
A neurofeedback claim has two halves — the participant learned to change the signal and something else
changed because of it — and both are routinely supported by an analysis that cannot support them: a trend
test on per-block values that are strongly autocorrelated. nb-7-3 measured that test's false-positive rate on
replayed recordings where learning is impossible by construction and found 8 of 12 sessions "significant".
This notebook reproduces that count through the course library, runs the four sham conditions through the
same loop, shows the valid session-level test being null where it should be null and powerful where a
rise is planted, and assembles a session log that says which of the CRED-nf items one person can satisfy
alone and which they cannot.
- The four sham modes through the library's loop, on the same twelve replayed sessions as
nb-7-3, with the library'snaive_trendbesidehelpers_l7.within_session_trend— the veridical count is a keyed number and must equal the stored 8 of 12. - The valid test: one number per session, a sign-flip permutation across sessions; null on the replayed sessions, and its power on a planted increase is a keyed number.
- A session log with a sealed sham mode, unblinded only with the seed; a pre-registration hashed before the session; the six CRED-nf items marked satisfied, not satisfiable alone, or unsatisfied.
Every number comes from replayed ds-eegbci data or the synthetic stream; no headset produced anything here.
# Setup: dependencies, the shared helpers, the course library, non-interactive plotting, a quiet downloader.
import importlib.util
import json
import logging
import subprocess
import sys
import tempfile
import time
import warnings
from pathlib import Path
_needed = ("mne", "scipy", "matplotlib", "pandas", "pooch")
_missing = [p for p in _needed if importlib.util.find_spec(p) is None]
if _missing:
_req = next((d / "requirements.txt" for d in (Path.cwd(), *Path.cwd().parents)
if (d / "requirements.txt").exists()), None)
_cmd = [sys.executable, "-m", "pip", "install", "-q"]
_cmd += ["-r", str(_req)] if _req else ["mne==1.10.2", "pooch>=1.8"]
subprocess.check_call(_cmd)
_shared = next((d / "_shared" for d in (Path.cwd(), *Path.cwd().parents)
if (d / "_shared" / "helpers_l7.py").exists()), None)
if _shared is None:
# No checkout around this notebook -- a downloaded .ipynb, or a fresh Colab kernel. Fetch the shared
# helpers the site publishes beside it. With a checkout present the walk-up above wins and nothing
# is downloaded; the modules land in ./_shared/ and are reused on every later run.
import json as _json
import urllib.request as _urllib
_base = "https://eeg.neurokinetikz.com/notebooks/_shared/"
_shared = Path.cwd() / "_shared"
_shared.mkdir(exist_ok=True)
for _name in _json.loads(_urllib.urlopen(_base + "index.json").read())["modules"]:
if not (_shared / _name).exists():
_urllib.urlretrieve(_base + _name, _shared / _name)
sys.path.insert(0, str(_shared))
import helpers_l7 as L7
_loop = L7.import_eegloop() # loop/eegloop on sys.path from any directory of a checkout; returns loop/
import helpers
import helpers_l7 as L7
import eegloop
from eegloop.analysis import crednf_report, learning_test, naive_trend, session_change
from eegloop.feedback import CREDNF_ITEMS, SHAM_MODES, FixedBaseline, ShamPolicy, SignalSpec, build_chain, combine
from eegloop.probe import run_offline
from eegloop.session import SessionReader, resolved, run_protocol, validate_protocol
from eegloop.sources import SyntheticSource, blocks
from eegloop.steps import Chain
from eegloop.steps.causal_filter import CausalFIR, fir_taps
from eegloop.versions import config_hash
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import mne
from scipy import stats
mne.set_log_level("WARNING")
logging.getLogger("pooch").setLevel(logging.WARNING)
warnings.filterwarnings("ignore", category=RuntimeWarning)
plt.rcParams["figure.dpi"] = 72
SOURCE = "replay"
LOCAL_RECORDING = None # a path to a recording of your own: a second pass on it, never shipped
# nb-7-3's recipe, exactly: twelve subjects, the first 60 s of O1 of R01 (eyes open) and R02 (eyes closed), µV
FS = float(L7.LATENCY_SPEC["sfreq_hz"]); BAND = tuple(L7.LATENCY_SPEC["band_hz"]); N_TAPS = int(L7.LATENCY_SPEC["fir_taps"]); BLOCK = int(L7.LATENCY_SPEC["block_samples"])
SUBJECTS = [f"S{i:03d}" for i in range(1, 13)]
SEGMENT_S = 60.0
NB73_STORED = {"veridical": 8, "sham-band": 6, "sham-yoked": 6, "sham-inverted": 8} # nb-7-3-realtime, final cell, item [6]
_edf_before = set(Path(mne.get_config("MNE_DATA", str(Path.home() / "mne_data"))).rglob("*.edf"))
print(f"eegloop {eegloop.__version__} · mne {mne.__version__} · pipeline {FS:g} Hz, {BAND} Hz, {N_TAPS} taps, {BLOCK}-sample blocks")
1 · The four sham modes, and the naive test's false-positive rate, reproduced¶
nb-7-3 replayed twelve eyes-closed minutes through the loop and ran the naive test — the OLS slope of the
per-block feedback on time — under each condition. Learning is impossible on a replayed recording, so every
"significant" session is a false positive. The library's naive_trend is that test ported verbatim and
labelled valid: False; the library's loop is the one L7.13 held to helpers_l7 to 10⁻¹². So the counts must
come out the same, and the veridical count must be the stored 8 of 12.
t0 = time.time()
eyes_open, eyes_closed = {}, {}
n = int(SEGMENT_S * FS)
for s in SUBJECTS:
for run, store in (("R01", eyes_open), ("R02", eyes_closed)):
raw = helpers.load_spine("ds-eegbci", s, [run], verbose=False).pick(["O1"])
assert raw.info["sfreq"] == FS
store[s] = raw.get_data()[0][:n] * 1e6
print(f"{len(SUBJECTS)} subjects × 2 runs, {SEGMENT_S:.0f} s of O1 each at {FS:g} Hz ({time.time() - t0:.1f} s)")
taps = fir_taps(N_TAPS, BAND, FS)
taps_control = fir_taps(N_TAPS, (16.0, 20.0), FS)
def loop_values(x, kernel=taps):
r = run_offline(Chain([CausalFIR(kernel, 1)]), x, fs=FS, block_samples=BLOCK)
return r["values"], r["times_s"]
# every subject's veridical values once; the sham modes derive from them exactly as the library's ShamPolicy does
ver = {s: loop_values(eyes_closed[s]) for s in SUBJECTS}
band = {s: loop_values(eyes_closed[s], taps_control) for s in SUBJECTS}
donor_of = {s: SUBJECTS[(i + 1) % len(SUBJECTS)] for i, s in enumerate(SUBJECTS)}
donor_vals = {s: loop_values(eyes_open[donor_of[s]])[0] for s in SUBJECTS}
def shown(mode, s):
v, t = ver[s]
pol = ShamPolicy(mode, donor=donor_vals[s] if mode == "sham-yoked" else None, control_band=(16.0, 20.0), seed=L7.SEED)
if mode == "sham-band":
return band[s][0], t
# the library's inverted sham reflects about the CALIBRATION baseline (the centre that exists when the value is shown);
# helpers_l7 reflects about the whole-session mean, which needs the future. Recorded, not reconciled (loop/README.md).
centre = FixedBaseline.from_values(v[:int(10 * FS / BLOCK)]).centre if mode == "sham-inverted" else None
return np.asarray([pol.value(float(vk), k, centre) for k, vk in enumerate(v)]), t
modes = ("veridical", "sham-band", "sham-yoked", "sham-inverted")
counts, rows = {}, []
for m in modes:
ps, ps_theirs, rising, lags = [], [], [], []
for i, s in enumerate(SUBJECTS):
v, t = shown(m, s)
ours = naive_trend(v, t)
theirs_r = L7.sham_feedback(eyes_closed[s], donor=eyes_open[donor_of[s]], mode=m)
theirs = L7.within_session_trend(theirs_r["values"], theirs_r["times_s"])
ps.append(ours["p"]); ps_theirs.append(theirs["p"]); rising.append(ours["slope_per_min"] > 0); lags.append(ours["lag1_autocorrelation"])
assert ours["valid"] is False
ps, ps_theirs = np.asarray(ps), np.asarray(ps_theirs)
counts[m] = int((ps < 0.05).sum())
rows.append({"condition": m, "naive test p < .05 (eegloop)": f"{counts[m]}/12", "p < .05 (helpers_l7)": f"{int((ps_theirs < 0.05).sum())}/12",
"and rising": f"{int(((ps < 0.05) & np.asarray(rising)).sum())}/12", "median lag-1 r": round(float(np.median(lags)), 3),
"max |Δp| ours vs helpers": f"{float(np.max(np.abs(ps - ps_theirs))):.1e}", "nb-7-3 stored": f"{NB73_STORED[m]}/12"})
tab = pd.DataFrame(rows)
print(tab.to_string(index=False))
print(f"\nnominal false-positive rate at alpha = .05: {0.05 * 12:.1f} of 12; the test is invalid, not unlucky (the last-but-two column is why)")
assert counts["veridical"] == NB73_STORED["veridical"], "the veridical count does not reproduce nb-7-3's stored 8 of 12"
for m in modes:
assert counts[m] == NB73_STORED[m], f"{m}: {counts[m]} vs stored {NB73_STORED[m]}"
print("every count reproduces nb-7-3's stored numbers through the library (the inverted sham's anchor differs and the OLS p-value is invariant to it)")
one = SUBJECTS[0]
fig, ax = plt.subplots(figsize=(12, 3.8))
for m, c in zip(modes, ("C0", "C1", "C2", "C3")):
v, t = shown(m, one)
ax.step(t, v, where="post", lw=1.0, color=c, label=f"{m} ({SHAM_MODES[m].split('.')[0][:60]}…)")
ax.set(xlabel="Time (s)", ylabel="Shown value (µV RMS per block)", title=f"{one}: what the participant would see under each condition; every one of them looks like feedback")
ax.legend(fontsize=7, loc="upper right"); plt.tight_layout(); plt.show()
2 · The valid test: the session is the unit¶
One number per session — the change from its first half to its second, as a fraction of the first — and then a
sign-flip permutation test across sessions (exact below thirteen sessions: every sign pattern is enumerated).
Nothing inside a session can inflate it, because the autocorrelation that broke the naive test lives inside a
session. nb-7-3 ran a session-level t-test on the same sessions; both are printed.
Read the yoked row before believing the word "null". A valid test detects a real change in the signal, and the yoked sessions — twelve eyes-open minutes, replayed as someone else's feedback — do change over a minute. That is the passage of time, not learning, and no within-condition test can tell the two apart. The claim needs the contrast between conditions: the veridical change minus the yoked change, session by session, which is what the last line computes.
rows = []
for m in modes:
sess = [shown(m, s)[0] for s in SUBJECTS]
lt = learning_test(sess, alternative="two-sided")
d = np.asarray([session_change(v) for v in sess])
t_s, p_s = stats.ttest_1samp(d, 0.0)
rows.append({"condition": m, "mean change (fraction)": round(lt["mean_change"], 4), "learning_test p (exact, two-sided)": round(lt["p"], 3),
"session-level t-test p (nb-7-3's)": round(float(p_s), 3), "n_null": lt["n_null"]})
assert lt["valid"] and lt["exact"]
valid_tab = pd.DataFrame(rows)
print(valid_tab.to_string(index=False))
d_ver = np.asarray([session_change(shown("veridical", s)[0]) for s in SUBJECTS])
d_yok = np.asarray([session_change(shown("sham-yoked", s)[0]) for s in SUBJECTS])
diff = d_ver - d_yok
signs = np.array(np.meshgrid(*([[-1.0, 1.0]] * diff.size))).T.reshape(-1, diff.size)
null = (signs * np.abs(diff)[None, :]).mean(axis=1)
p_contrast = float(np.mean(np.abs(null) >= abs(diff.mean()) - 1e-15))
t_c, p_tc = stats.ttest_rel(d_ver, d_yok)
print(f"\nthe controlled contrast, veridical change minus yoked change per subject: mean {diff.mean():+.4f}, exact sign-flip p = {p_contrast:.3f} (nb-7-3's paired t-test p = {p_tc:.3f})")
yoked_rise = float(valid_tab.loc[valid_tab.condition == "sham-yoked", "learning_test p (exact, two-sided)"].iloc[0])
print(f"\nthe yoked sessions rise within the minute (p = {yoked_rise:.3f}): a real change in the eyes-open donors' alpha over sixty seconds, and not learning —")
print("which is exactly why a claim needs the contrast between conditions, not a within-condition test, however valid")
3 · Power on a planted increase¶
A test that is null where it should be is only half of validity; it also has to detect a change that exists. Twelve synthetic sessions, each with the same planted linear rise in the 8–12 Hz signal, and the valid test run over thirty replicates; the same with no rise, for the false-positive rate. The power at the planted rise is the lesson's keyed number.
def synthetic_sessions(rise, seed0, n_sessions=12, duration_s=120.0):
out = []
for k in range(n_sessions):
src = SyntheticSource("alpha-schedule", duration_s=duration_s, alpha_uv=6.0, alpha_bursts=(), seed=seed0 + k, pace="fast")
spec = SignalSpec(band=BAND, channels=("TP9", "TP10"), n_taps=N_TAPS)
chain = build_chain(spec, src.info, 32, quality=None)
picks = spec.picks(src.info.ch_names)
t, v = [], []
for blk in blocks(src, 32):
t.append(blk.t_end_s); v.append(combine(chain.process(blk).flags["envelope"], picks, "rms"))
t, v = np.asarray(t), np.asarray(v)
out.append(v * (1.0 + rise * np.clip((t - 10.0) / (duration_s - 10.0), 0, 1)))
return out
t0 = time.time()
N_REP = 25
power = {}
for rise in (0.0, 0.05, 0.10, 0.30):
ps, naive = [], []
for rep in range(N_REP):
sess = synthetic_sessions(rise, (1000 if rise else 5000) + 100 * rep)
ps.append(learning_test(sess, seed=rep)["p"])
naive.append(np.mean([naive_trend(v, np.arange(v.size) * 32 / 256.0)["p"] < 0.05 for v in sess]))
power[rise] = (float(np.mean(np.asarray(ps) < 0.05)), float(np.mean(naive)))
print(f"{N_REP} replicates of 12 synthetic sessions each ({time.time() - t0:.0f} s):")
print(f" {'planted rise':>14s} {'learning_test p < .05 (power)':>32s} {'naive test: sessions flagged':>30s}")
for rise, (pw, nv) in power.items():
print(f" {'none' if rise == 0 else f'+{rise * 100:.0f} %':>14s} {pw * 100:31.0f} % {nv * 100:29.0f} %")
fpr = power[0.0][0]
print(f"\nwith no rise the valid test fires in {fpr * 100:.0f} % of replicates (nominal 5 %); the naive test flags sessions whatever the rise, because its p-value is broken, not because it is sensitive")
4 · A session log with a sealed sham mode, and the pre-registration that precedes it¶
The library's session runner writes what CRED-nf asks for: the signal in full, the budget with its measured row, every value the participant saw, every gate closure and reward, the protocol with every decision filled in, and the sham mode sealed into the log as a token only the seed unblinds. A pre-registration is a dictionary hashed before the first session; whether the hash was committed before the data is a fact about version control, not about the file, and the report says so. Run a veridical session on the synthetic stream as a donor, then a yoked session that replays it, and read both back.
PROTO = {
"name": "alpha-up", "seed": 20260920, "block_samples": 32, "processing_ms": 10.0,
"source": {"kind": "synthetic", "scenario": "alpha-schedule", "pace": "fast"},
"signal": {"band": list(BAND), "channels": ["TP9", "TP10"], "n_taps": N_TAPS},
"quality": {"enabled": True, "window_s": 1.0, "hold_s": 0.5, "mains_hz": 60},
"baseline": {"mode": "fixed"}, "reward": {"mode": "continuous", "lo_z": -1.0, "hi_z": 2.0},
"phases": [{"kind": "calibrate", "duration_s": 20}, {"kind": "train", "duration_s": 100}],
}
prereg = {
"outcome": "second-half minus first-half mean of the raw per-block 8–12 Hz RMS at TP9+TP10, as a fraction of the first half, per session",
"test": "eegloop.analysis.learning_test, sign-flip permutation across sessions, alternative 'greater', alpha 0.05",
"baseline": "fixed from the 20-s calibration; the display's threshold is not the outcome",
"control": "sham-yoked, sealed; unblinded after the analysis with the protocol seed",
"protocol_hash": config_hash(resolved(validate_protocol(PROTO, source="mem"))),
}
prereg_hash = config_hash(prereg)
print("pre-registration hash:", prereg_hash, "(what a commit before the first session would carry)")
with tempfile.TemporaryDirectory() as tmp:
donor = run_protocol(validate_protocol(PROTO, source="mem"), out_dir=Path(tmp) / "donor")
yoked_proto = dict(PROTO, name="yoked", source=dict(PROTO["source"], seed=7), sham={"mode": "sham-yoked", "donor": str(Path(tmp) / "donor")})
yoked = run_protocol(validate_protocol(yoked_proto, source="mem"), out_dir=Path(tmp) / "yoked")
rd, ry = SessionReader(Path(tmp) / "donor"), SessionReader(Path(tmp) / "yoked")
print(f"\ndonor session: {donor.status}, {donor.stats['n_blocks']} blocks, {donor.stats['n_gated']} gated; sealed sham token {rd.sealed_sham}")
print(f"yoked session: {yoked.status}, {yoked.stats['n_blocks']} blocks; sealed sham token {ry.sealed_sham}")
for name, r in (("donor", rd), ("yoked", ry)):
print(f" {name}: unblinded with the seed → {ShamPolicy.unblind(r.sealed_sham, PROTO['seed'])!r}")
nmin = min(ry.signal["shown_values"].size, rd.feedback[1].size)
print(f" the yoked participant saw the donor's values block for block: max |Δ| {float(np.max(np.abs(ry.signal['shown_values'][:nmin] - rd.feedback[1][:nmin]))):.1e} µV")
print("\nfiles in a session directory:", sorted(p.name for p in (Path(tmp) / "yoked").iterdir()))
print("the signal, as logged:", ry.session["signal"])
print("\nCRED-nf, from the yoked session's log alone:")
for item in crednf_report(ry.protocol, ry.session):
print(f" [{item['status']:22s}] {item['item'][:70]}…")
print("\n…and with the pre-registration committed and the sessions tested:")
report = crednf_report(ry.protocol, ry.session, prereg_committed=True,
sessions_tested=learning_test([shown('veridical', s)[0] for s in SUBJECTS]))
for item in report:
print(f" [{item['status']:22s}] {item['item'][:70]}…")
n_alone = sum(1 for i in report if i["status"] == "satisfied")
statuses = {i["status"] for i in report}
The report never ticks a box the log cannot vouch for. The sealed mode makes participant blinding possible; whether the experimenter and the analyst were blind is a fact about people. A single session cannot show learning; several sessions and the valid test can, and the result says nothing about any outcome — that is a separate claim with separate evidence, which is the lesson's first sentence restated as a table.
Your own recording, if you have one¶
When LOCAL_RECORDING names a file — a .bin with its sidecar as eegloop run --record-raw writes it, or an
.npz — the cell below runs this notebook's measurement on it and prints the result titled as yours; when it
is None the cell is skipped entirely, so nothing empty ships. Nothing from your recording is written into
this notebook's outputs on the site: it is a second pass, never shipped (CONTRACTS, Phase 5).
if LOCAL_RECORDING is not None:
mine = ReplaySource(LOCAL_RECORDING, pace="fast", block_samples=32)
dur = mine.duration_s
proto_mine = {"name": "mine", "seed": 20260920, "block_samples": 32, "processing_ms": 10.0,
"source": {"kind": "replay", "path": str(Path(LOCAL_RECORDING).name), "pace": "fast"},
"signal": {"band": [8.0, 12.0], "channels": [], "n_taps": 129},
"quality": {"enabled": True, "window_s": 1.0, "hold_s": 0.5, "mains_hz": 60},
"baseline": {"mode": "fixed"}, "reward": {"mode": "continuous", "lo_z": -1.0, "hi_z": 2.0},
"sham": {"mode": "sham-inverted"},
"phases": [{"kind": "calibrate", "duration_s": min(20.0, dur / 3)}, {"kind": "train", "duration_s": max(1.0, dur - min(20.0, dur / 3) - 1.0)}]}
with tempfile.TemporaryDirectory() as tmp_mine:
res_mine = run_protocol(validate_protocol(proto_mine, source="mem"), source=mine, out_dir=Path(tmp_mine) / "mine")
rm = SessionReader(Path(tmp_mine) / "mine")
print(f"your recording — not shipped: session {res_mine.status}, {res_mine.stats['n_blocks']} blocks, {res_mine.stats['n_gated']} gated; sealed token {rm.sealed_sham}")
print("unblinded with the seed:", repr(ShamPolicy.unblind(rm.sealed_sham, proto_mine["seed"])))
for item in crednf_report(rm.protocol, rm.session):
print(f" [{item['status']:22s}] {item['item'][:70]}…")
else:
print("LOCAL_RECORDING is None: skipped (set it to a .bin with its sidecar, or an .npz, to run a sealed-sham session over your own file)")
5 · The numbers the lesson's exercises ask for¶
print("nb-7-14-protocol — L7.14 numbers (draft; TODO(confirm) at author review)")
print()
print(f"(a) Of 12 replayed veridical sessions where learning is impossible, the naive test called {counts['veridical']} significant "
f"(nb-7-3 stored 8; sham-band {counts['sham-band']}, sham-yoked {counts['sham-yoked']}, sham-inverted {counts['sham-inverted']}); nominal 0.6.")
print(f"(b) Power of learning_test on 12 synthetic sessions, {N_REP} replicates: " + "; ".join(f"+{r * 100:.0f} % rise → {pw * 100:.0f} %" for r, (pw, _) in power.items() if r) +
f"; false-positive rate with no rise {fpr * 100:.0f} %. On the replayed sessions: veridical p = {valid_tab.iloc[0]['learning_test p (exact, two-sided)']:.3f}, "
f"yoked p = {yoked_rise:.3f} (a real within-minute change, not learning), controlled contrast p = {p_contrast:.3f}.")
print(f"(c) Six CRED-nf items; from one session's log alone the report marks 3 satisfied and 3 not satisfiable alone; with a committed pre-registration "
f"and the sessions tested, {n_alone} of 6 are satisfied and the remaining one (blinding of experimenter and analyst) is about people, not files.")
Cleanup¶
Only what this run fetched is deleted; the session logs were written to a temporary directory.
_edf_after = set(Path(mne.get_config("MNE_DATA", str(Path.home() / "mne_data"))).rglob("*.edf"))
_new = sorted(_edf_after - _edf_before)
for p_ in _new:
p_.unlink(missing_ok=True)
assert len(eyes_closed) == len(SUBJECTS)
print(f"deleted {len(_new)} EDF file(s) this run fetched; kept everything that was already cached; cohort {len(SUBJECTS)}/{len(SUBJECTS)}")