Decoding and BCI

Why this matters

A BCI paper reports 82 % on four-class motor imagery. Another reports 74 % with a method it calls better. Neither number means anything on its own, because they are not answers to the same question: one may be within-session, the other across days; one may pool sessions into a random split, the other hold one out; one may count a subject the other excluded. A benchmark is a protocol before it is a result, and most disagreements in this literature are disagreements about the protocol.

This lesson runs three decoding pipelines under one protocol, on two paradigms, and reads the results the way L6.5 says a decoding result must be read. It is also the lesson where the dataset’s own documentation forbids two things people routinely do to it.

Concepts

What a benchmarking framework actually fixes

moabb exists because the comparison above is hard for boring reasons: every dataset has its own file format, its own event codes, its own trial windows and its own idea of what a “run” is. The framework separates four things that papers usually tangle:

• the dataset — the recording and its events;
• the paradigm — what counts as a trial, which classes, which band, which window. Two papers on “motor imagery” that use 0.5–2.5 s and 0–4 s after the cue are not testing the same thing;
• the pipeline — preprocessing, features, classifier, as one fitted object;
• the evaluation — how the data are split: within session, across sessions of the same subject, or across subjects.

The last of those is where most of the disagreement lives, and it is the one the framework makes impossible to leave unstated. Swapping the evaluation while holding everything else fixed usually moves the number more than swapping the classifier does — which is the lesson’s real finding and the reason its exercise asks about the cross-subject margin rather than the headline accuracy.

TODO(confirm): the exact class names moabb uses for its dataset, paradigm and evaluation objects were not verified against moabb’s own documentation in this repository. The dataset entry records the loader as moabb.datasets.BNCI2014_001() and was read from moabb 1.7.2’s source (site/notes/data-p4-licences.md §3.1); the evaluation classes are described here by what they do, not by an API name this site has checked.

Three pipelines, and what each one assumes

CSP + LDA. Fit spatial filters that maximise the variance ratio between two classes, take the log variance of a handful of filtered components as features, and classify them linearly (L5.7). It assumes the classes differ in band-limited power with a fixed spatial distribution — which is exactly what sensorimotor rhythm changes are, and exactly what a P300 is not. Multi-class CSP needs an extension (one-versus-rest, or a joint diagonalization); say which one you used.

Riemannian classifiers. Take the trial covariance matrix as the feature and stop trying to turn it into a vector first. Covariance matrices live on a curved space, not in a flat one, and the distances that respect that geometry behave much better under the shifts that plague EEG — changed impedance, a different day, a different head. Two common forms: classify by distance to each class’s geometric mean, or project the covariances onto the tangent space at the overall mean and run an ordinary linear classifier there ( (Barachant et al., 2012) ). The second is a drop-in feature transform, which is why it appears in so many pipelines.

EEGNet. A compact convolutional network designed so that the number of parameters stays in proportion to the number of trials EEG actually provides, rather than to the number a computer-vision dataset provides ( (Lawhern et al., 2018) ). It learns its temporal filters and its spatial filters instead of being given them, which is its appeal and also its cost: it has more ways to fit the noise, and its regularization, its early stopping and its epoch budget are all hyperparameters that must be chosen inside the fold (L6.5).

TODO(confirm): EEGNet’s layer structure and its default hyperparameters are not restated here because they were not read from (Lawhern et al., 2018) or from braindecode’s source in this repository. The notebook prints the architecture and the training settings it actually used; quote those, not a remembered summary.

The honest expectation, which (Lotte et al., 2018) and (Jayaram, 2018) both support in outline, is that these three are close on most datasets and that the ordering changes with the paradigm, the trial count and the evaluation. A benchmark that finds a large, consistent, cross-subject win for one of them has found something unusual and should be checked for the reason before it is celebrated.

Within-session, cross-session, cross-subject

These are three different claims, and the numbers fall as you go down the list.

• Within session. Hold out trials of the same session. It answers “could this decoder work after a calibration block today?” — the question a practical BCI with a calibration phase actually asks.
• Cross session. Train on one day, test on another. Everything that drifts between days — electrode positions, impedance, the participant’s strategy, the participant’s alertness — now counts against you. This is the honest estimate for a system a person would use more than once.
• Cross subject. Train on other people, test on a new one. This is the “calibration-free” claim, and it is a different and much harder problem, because a decoder must now be right about what is shared across heads rather than about one head.

Transfer is the family of methods for closing that last gap: aligning each subject’s covariances to a common reference before pooling, re-centring the feature distribution, or fine-tuning a network trained on others with a little data from the new person. Transfer moves a cross-subject number towards a within-subject number, so a paper that reports “cross-subject” accuracy after fine-tuning on the target subject is reporting neither, unless it says exactly how much target data it used.

The dataset says two things about itself that most analyses ignore

ds-bci-iv-2a is nine subjects, 22 EEG channels plus 3 EOG at 250 Hz, four imagery classes — left hand, right hand, both feet, tongue — with 288 trials in each of two sessions.

Its EOG channels must not be used for classification. The dataset’s own description document says so in as many words: “The EOG channels are provided for the subsequent application of artifact processing methods and must not be used for classification.” This is not a licence term and not a stylistic preference: it is the recording group telling you that those three channels carry the eye movements the paradigm cannot avoid, and that a classifier fed them will decode the eyes. A pipeline that selects channels by name and forgets to exclude them does the forbidden thing silently and scores better for it, which is pf-decoding-leakage in its purest form — the information is real, the accuracy is real, and the claim is false.

Its two sessions are different days. They are labelled T and E, they were recorded in separate sessions, and the evaluation labels were released only after the competition deadline. Train on T and test on E and you have a genuine cross-session evaluation. Pool the 576 trials and take a random five-fold split and you have leaked the session: every test fold is surrounded by training trials from the same day, the same cap placement and the same impedances, and the score rises for a reason that has nothing to do with the classifier. The number that comes out is not wrong about the data; it is an answer to a question nobody asked.

Two more facts worth holding on to: the signals were band-passed 0.5–100 Hz with a 50 Hz notch enabled at acquisition, so this is not unfiltered data the way ds-eegbci is, and anything taught on it about line noise or drift is being taught through an analog chain somebody else chose; and its own competition rules required that “All algorithms must be causal”, which is the constraint L7.3 spends a whole lesson on.

Nothing derived from this dataset ships on this site

ds-bci-iv-2a is licensed CC BY-ND 4.0 — attribution, no derivatives — stated by the recording group itself (the Institute for Knowledge Discovery at Graz) on the BNCI Horizon 2020 database page that moabb downloads from, and corroborated by moabb’s own metadata. Every snippet this site ships is a derivative: cropped, filtered, re-referenced. So:

• the notebook may download it, compute on it and publish its accuracies, because analysing data is not redistributing it, and a results table is a report of findings;
• the site may not ship a trace, a figure, a precomputed CSP product or any other asset built from its signal, and it does not.

If you want to reproduce anything in this lesson on this dataset, you fetch it yourself — which the notebook does for you, per subject, from the channel the licence was read on. That is a real cost of the licence and it is stated rather than worked around. The full record, with every statement read and where it was read, is in the dataset entry.

The second paradigm, and why accuracy is the wrong word for it

ds-brain-invaders (moabb’s BI2014a) is a P300 speller-style task recorded with 16 dry electrodes on 64 subjects at 512 Hz — the same decoding question asked of hardware that behaves quite differently (L7.5). It is CC BY 4.0 and open, so unlike the motor-imagery set it could ship assets.

Its classes are wildly unbalanced by design: roughly 198 target flashes against 990 non-targets per subject, about one in six. A classifier that answers “non-target” every time scores 83 % and has learned nothing. Report the area under the ROC curve, or balanced accuracy, and say which — the same point L6.5 makes about chance levels, with the arithmetic already done for you.

Reading a benchmark table

Four habits, all of which the notebook’s output is arranged to support:

1. Read the spread before the mean. Nine subjects is nine numbers. A method that wins on the mean while losing on five of nine subjects has won an argument about outliers.
2. Put chance on the plot, with its band. Four balanced classes means chance is 25 %, not 50 %. On the 288 trials of one evaluation session the 95 % band around chance runs from 20.0 % to 30.0 % on the normal approximation — so a subject at 30 % is not evidence of anything.
3. Compare like with like. A cross-session number and a within-session number from the same table are two different quantities; putting them in adjacent columns is fine, putting them in the same bar chart is not.
4. Say what was tuned and where. Every hyperparameter chosen by looking at a score needs an inner loop, and a benchmark is a very large search space.

The data behind this lesson

• The widget below is built on ds-eegbci (ODC-By 1.0, open, snippets allowed), not on ds-bci-iv-2a, for the licence reason above. Its numbers — 86.7 % honest against 97.8 % leaked on one subject, and a twelve-subject spread of 44.4–95.6 % with a median of 55.6 % — come from data/scripts/make_csp_products.py and are the same numbers L6.5 quotes, including its open disagreement about whether that median is 55.6 % or 60.0 %, which is recorded there and in pf-decoding-leakage rather than resolved. The shipped subject was selected by accuracy under a stated rule, and that selection is part of the number.
• The notebook downloads ds-bci-iv-2a and ds-brain-invaders itself and ships nothing back.
• No figure derived from ds-bci-iv-2a appears anywhere on this site, for the reason given above.

Explore

The widget runs on ds-eegbci, which is the dataset this site is allowed to ship. Four things to read, in this order:

• The subject survey first, the winner second. Twelve bars, one per candidate subject, with the shipped one marked. Eight of the twelve sit inside the chance band for 45 epochs. A benchmark table is this picture with the bars hidden.
• The leakage control. Fit the spatial filter on all the trials instead of inside each fold and watch the accuracy rise by eleven points with nothing about the decoder improved. Now imagine that control being a line of code you did not write.
• The fold-by-fold scores, not the mean. With a few dozen trials per class a single fold moves the mean by several points, which is what a benchmark’s error bars are made of.
• The feature-set list. CSP with two components scores below plain band power at C3 and C4. More components is not monotonically better, and choosing the component count by the score is a search you must pay for.

Practice

The notebook runs CSP + LDA, a Riemannian tangent-space pipeline and EEGNet on ds-bci-iv-2a motor imagery and on ds-brain-invaders P300, under one paradigm definition each, with within-session, cross-session and cross-subject evaluations, and prints a results table for both paradigms. It downloads per subject and deletes what it fetched, drops the three EOG channels explicitly and says why, and never pools T and E into a random split. Its final cells print the cross-subject means, the per-subject spread and the margin between the best and second-best pipeline — the numbers the first two exercises ask for.

Exercises

Pitfalls