Level 7 L7.14 judgment thread

A protocol that can fail: sham, blinding, pre-registration, the valid test

The block-level trend test is anti-conservative and its false-positive rate is measured — eight of twelve replayed sessions "significant" where learning is impossible. The four sham conditions and what each does and does not control; a valid test with the session as the unit, null where it should be and powerful where a rise is planted, and why even a valid within-condition test cannot separate learning from the passage of a minute; a pre-registration hashed before the data; a session log with a sealed sham mode; the CRED-nf items one person can satisfy alone and the ones they cannot.

~60 min Notebook: nb-7-14-protocol

Prerequisites: L7.13 · The feedback signal: band power in a buffer, L6.4 · Circularity and analytic flexibility, L6.6 · Reporting standards

What you will be able to do

  • Explain why the block-level trend test is anti-conservative and show its false-positive rate
  • Choose a sham mode and state what it does and does not control
  • Write a pre-registration for one learning outcome
  • Assemble a session log that satisfies the CRED-nf items you can satisfy alone and names the ones you cannot

Why this matters

L7.3 separated the two claims a neurofeedback study makes — the participant learned to change the signal and something else changed because of it — and named the controls the second needs: a sham condition, blinding, a pre-registered outcome. This lesson is about the first claim, which is the one an application developer can actually test, and about the analysis that is routinely used to test it and cannot: a trend across the per-block feedback values of a session. Those values are strongly autocorrelated, so an ordinary regression’s standard error is far too small and its p-value is anti-conservative — and nb-7-3 measured how much, on twelve replayed recordings where learning is impossible by construction: eight of twelve sessions “significant”. This lesson reproduces that count through the course library, so the number is not a memory; puts the four sham conditions through the same loop; shows a valid test that is null where it should be and powerful where a rise is planted; and then makes the point the valid test alone cannot make, which is that a within-condition change is not learning until it is compared with a control. It ends with the library’s session log, which seals the sham mode, and the report that says which CRED-nf items a single person with a single headset can satisfy — and which they cannot, however carefully they work.

Concepts

Two claims, and the one you can test

“The participant learned to raise the signal” is a claim about the signal, and it can be tested from the session logs. “The participant’s attention, sleep, or symptom improved because of it” is a claim about an outcome, and it needs a control group, blinding and an outcome measured by someone who does not know the condition — the design L6.6 and the CRED-nf checklist describe. Everything in this lesson serves the first claim and keeps the second separate: the library’s crednf_report marks “evidence that the participants learned to change the targeted signal” as its own item, reported separately from whether the clinical or behavioural outcome changed.

What this lesson connects to

No device is named. Every replayed session here is ds-eegbci — twelve subjects, the first sixty seconds of O1 in the eyes-closed run, through the L7.3 pipeline at 160 Hz — because that is the recipe nb-7-3 used and the point is to reproduce its numbers through the library rather than to produce new ones. The synthetic sessions are the library’s frontotemporal-4 stream with a planted rise. A headset would produce session logs of exactly the shape read here, and nothing about the tests depends on which one.

The naive test, and its false-positive rate

The naive test is the OLS slope of the feedback value on time within a session: “did it go up?” The library ships it as naive_trend, ported verbatim from the notebook helpers and labelled valid: False, because the course shows what it does wrong rather than pretending it does not exist. On twelve replayed eyes-closed minutes — data in which no learning can occur, because the participant is a file — it calls 8 of 12 veridical sessions significant at α = .05 against a nominal 0.6; the sham conditions flag 6, 6 and 8. The library reproduces every count of nb-7-3 to the last digit (the two implementations’ p-values agree to 10⁻¹⁶), and the cause is in the same column: a median lag-1 autocorrelation of 0.69 in the per-block values. The test is invalid, not unlucky, and the sham rows show that “alpha went up during the session” carries no information about whether the feedback was real: it does the same thing when the feedback came from somebody else.

Four sham conditions, and what each controls

The library carries the four control conditions the notebook helpers name, verbatim, and runs each through the loop:

  • veridical — the participant’s own band-limited signal, in real time; the experimental condition;
  • sham-yoked — feedback replayed from a different recording on the same schedule. It looks and moves like feedback and carries no information about this participant’s brain, so it controls for the display, the task, time on task and expectation; it is the control CRED-nf asks for by name;
  • sham-band — the same participant, in real time, from a control band the protocol does not target. It controls for the participant’s own arousal and signal quality, which the yoked sham does not, and it does not control for the possibility that the control band moves with the target band;
  • sham-inverted — the target band, inverted about a centre. It controls for everything the veridical condition does except the direction of the contingency, and a participant who learns to lower the number is doing the same thing in reverse, which makes it hard to interpret.

One divergence between the library and the notebook helpers is recorded rather than reconciled: the helpers invert about the whole-session mean, which needs the future and so exists only offline; the library inverts about the calibration baseline, the centre that exists at the moment the value is shown. The naive test’s p-value is the same under either, because a reflection does not change a slope’s significance.

The valid test: the session is the unit

One number per session — the change from its first half to its second, as a fraction of the first — and then a sign-flip permutation test across sessions, exact below thirteen sessions because every sign pattern can be enumerated. Nothing inside a session can inflate it: the autocorrelation that broke the naive test lives inside a session, and the session is now a single observation. On the twelve replayed veridical sessions it is null (p = 0.59); on twelve synthetic sessions with a planted rise it detects a 5 % rise in 96 % of replicates and a 10 % rise in every one, with a false-positive rate of 4 % when nothing is planted — the naive test, on the same synthetic sessions, flags a quarter of them with no rise at all. learning_test answers only whether the targeted signal moved; it says nothing about any outcome.

Why a valid test on one condition is still not enough

Read the yoked row of the notebook’s table before trusting the word “null”. The yoked sessions — twelve eyes-open minutes, replayed as someone else’s feedback — do change over a minute: their alpha rises from the first half to the second, and the valid test says so (p = 0.005). That is a real change in the signal, and it is not learning; it is what sixty seconds of sitting with the eyes open does to posterior alpha. No within-condition test, however valid, can tell the two apart. The claim needs the contrast: the veridical change minus the yoked change, session by session, which comes out null (p = 0.16) — correctly, on data where the null is true. That is the whole reason the control condition exists, restated as a table, and it is why the pre-registration below names the contrast and not the trend.

A pre-registration, hashed before the data

A pre-registration for one learning outcome is a short file: the outcome (the change in the raw per-block signal against a baseline fixed before training, per session), the test (learning_test, the alternative, the α), the baseline rule, and the control condition and its sealing. The library hashes it the way it hashes a protocol, and the hash is what a commit before the first session would carry. Whether it was committed before the data is a fact about version control, not about the file, and the report says so rather than ticking the box: the item is not satisfiable alone from the log, and satisfied only when the person running the analysis vouches for the commit.

The session log, and the items one person can satisfy

The library’s session runner writes what the checklist asks for: the signal in full (item 4), every value the participant saw with the gate’s verdicts and rewards (item 6), the protocol with every decision filled in, the latency budget with its measured row, and the sham mode sealed into the log as a token only the protocol’s seed unblinds — so the participant can be blind, and so can the file. crednf_report reads a log and marks each of the six items satisfied, not satisfiable alone, or unsatisfied, with why. From one yoked session’s log alone: the control condition, the signal in full and the data are satisfied; pre-registration, blinding and learning are not satisfiable alone. With the pre-registration committed and several sessions tested, five of six are satisfied, and the sixth — blinding of the experimenter and the analyst — is about people, not files. A single self-run on your own headset supports no efficacy claim, and the report will keep saying so.

The data behind this lesson

ds-eegbci (ODC-By 1.0): S001–S012, the first sixty seconds of O1 in R02 (eyes closed) as the twelve sessions and in R01 (eyes open) as the yoked donors — nb-7-3’s recipe exactly, through the library’s loop. The library’s synthetic alpha-schedule stream with a planted linear rise for the power estimates, and as the source of the two logged sessions. No headset produced anything on this page.

Explore

This lesson has no widget; the session log is the artefact, and the notebook writes and reads one.

Practice

A protocol that can fail: the four sham modes through the library's loop on nb-7-3's twelve replayed sessions with the naive test's false-positive rate reproduced exactly, the valid session-level test null where it should be and the controlled contrast, its power on planted rises, a session log with a sealed sham mode unblinded only with the seed, a hashed pre-registration, and the six CRED-nf items marked satisfied or not satisfiable alone nb-7-14-protocol

Level 7 ~4 min
notebooks/L7/nb-7-14-protocol.ipynb

Downloads from ds-eegbci.

Open in Colab Download Read it here

Exercises

Exercise ex-7-14-naive-false-positives

Numeric

Twelve eyes-closed recordings are replayed through the feedback loop, so learning is impossible by construction. The naive within-session trend test — the OLS slope of the per-block feedback value on time — is run on each. How many of the twelve veridical sessions does it call significant at α = .05?

sessions

Exact answer required, in sessions.

Exercise ex-7-14-valid-power

Numeric

The notebook plants a linear rise of 5 % in the 8–12 Hz feedback signal across twelve synthetic sessions and runs the valid session-level test, repeating the whole experiment with 25 different seeds. In what percentage of the 25 replicates does the valid test reject the null at α = .05?

%

Accepted within ±8 %.

Exercise ex-7-14-crednf

Checklist (self-graded)

You have run ten sessions on your own headset with the sham mode sealed by the library, kept the session logs, and computed the learning outcome with the valid test. Tick each CRED-nf item that your logs and your actions can satisfy on their own, and leave unticked the ones that need someone else or something outside the files.

Checklist

0 / 6 checked. Self-graded.

Pitfalls

Pitfall

An adaptive threshold is a moving target

Symptom
Reward rate is the same in every session, whatever the signal did, and 'improvement' has no definition.
Cause

An adaptive threshold — re-fixed from the last thirty seconds, the last block of trials, or the previous session — keeps the participant near threshold by construction. That is its purpose: it makes the display feel responsive and keeps the reward rate in a band the protocol designer chose. But the reward rate is then a property of the rule, not of the signal. A participant whose alpha doubled an…

Detect
  • Read the protocol for the threshold rule. “Adjusted to maintain a reward rate of about x %” is the sentence. - Plot the reward rate across sessions. Flat at the target rate, whatever the signal did, is the signature. - Check whether the raw feature and the threshold were logged separately. If only the reward events survive, the outcome cannot be recovered. - Ask whether the pre-registered outco…
Fix
  • Separate the two jobs. Let the display use whatever rule trains best, and define the outcome on the raw feature against a baseline fixed once, before training, and never re-fixed. - Log the raw per-block value and the threshold as two columns (the library’s session log does), so the outcome can be computed after the fact even if the display rule changes. - Pre-specify the learning outcome on th…

Full entry with example →

Pitfall

Windows chosen from the tested data

Symptom
Effect only in the window that looked biggest.
Cause

A measurement window, an electrode set, a measure and a polarity are all selections. Making a selection from the same data you then test means the test no longer has the error rate it claims, because the selection has already searched over alternatives the test knows nothing about.

Detect
  • Read the methods for where the window came from. “Determined from the grand average of the condition difference” is the failure; “determined from the grand average collapsed across conditions” is not. - Count the choices you would have accepted, not the ones you made. If more than one window, electrode set or measure would have been defensible, the analysis has degrees of freedom the p-value do…
Fix
  • Fix the window in advance, from published work or from your own previous data, and record it before unblinding. - Use a collapsed localizer when the component’s latency in your sample is genuinely uncertain: choose the window and electrodes from the average of all conditions together, which under the null carries no information about the condition difference, and then measure the difference the…

Full entry with example →

Terms used here

Every term this lesson leans on, with the one-line meaning. Follow a term for its full entry.

Sham feedback
Feedback that looks and moves like the real thing but carries no information about the participant's target signal — the control a neurofeedback claim needs.
Pre-registration
The outcome, test and control condition written down and committed before any data exist, so the analysis cannot be chosen after seeing the result.

Reading

  1. Thibault et al. (2016). Self-regulating brain and neurofeedback. unverified
  2. Ros et al. (2020). CRED-nf checklist. unverified