Decoding as inference

Objectives

• Run time-resolved decoding with proper cross-validation
• Compute chance level and permutation-based significance
• Use temporal generalization
• State what above-chance decoding does and does not mean

Why this matters

Here is a decoding result from one subject of a motor-imagery dataset: 86.7 % correct, against a 50 % chance level, cross-validated. Here is the same pipeline on the same 45 epochs of the same subject with one line moved — the spatial filter fitted once on all the data instead of inside each training fold: 97.8 %. Eleven points, for free, from a change no reader of the methods section would notice. And here is the same pipeline on the other eleven candidate subjects: 44.4 % to 95.6 %, with a median near 56 %. The 86.7 % subject was chosen because it scored well.

Decoding is a statistical test with an unusually attractive output and an unusually large number of ways to be wrong. This lesson is about running it as inference.

Concepts

A classifier is a multivariate test statistic

Fit a classifier to discriminate two conditions from the pattern across channels, score it on data it has not seen, and the score is a statistic: high when the conditions differ in a way the classifier’s hypothesis class can express, at chance when they do not. That framing settles most of the interpretive questions later in the lesson. Accuracy is not an effect size, it is a test statistic; its value depends on the classifier, the features, the trial count and the noise, so it is not comparable across studies any more than an F ratio is.

The advantages over a univariate test are real: it uses all channels at once, so a difference that is distributed across the scalp without being large anywhere can be detected; it needs no a-priori electrode or window; and it produces one number per time point rather than per channel per time point.

Cross-validation, and where the fold boundary goes

Cross-validation estimates how well the fitted model performs on data it has not seen. The estimate is valid exactly to the extent that the held-out fold really is unseen — and “unseen” applies to the whole pipeline, not just the classifier.

Every step that looks at the labels must be fitted inside the fold. That includes:

• spatial filters — CSP, xDAWN, any supervised projection (L5.7);
• feature selection, channel selection, time-window selection;
• standardization and scaling, if the scaler’s parameters are estimated on all the data;
• class balancing or resampling;
• hyperparameter choice — which needs its own inner loop, hence nested cross-validation;
• and anything else chosen because it improved the score.

Steps that never see a label — a fixed band-pass, a fixed epoch window, a montage — can sit outside. The rule is not “preprocessing goes outside”; it is “label-aware goes inside”.

Two more boundaries that matter in EEG specifically:

• Trials that are not independent. Epochs from the same block, or overlapping epochs, or trials adjacent in time, share drift and artifact. Random k-fold splits them across the boundary and inflates the score; block-wise or run-wise splits do not. Where the data have a session or run structure, use it (GroupKFold and its relatives exist for this).
• Subjects. A within-subject decoder generalises to new trials of that subject. A claim about people needs the subject to be the held-out unit, which is a different and much harder problem, and usually a much lower number.

Chance is a distribution, not a number

With two balanced classes the expected accuracy under the null is 50 %. The observed accuracy under the null is not.

On 45 epochs, the 95 % binomial interval around chance runs from 35.4 % to 64.6 %. A single subject scoring 62 % is inside that band, and reporting it as “above chance” without the band is reporting noise. The interval narrows as the square root of the number of test trials, which is why single-subject decoding on a few dozen epochs is so much weaker evidence than the same percentage on thousands.

Three further cautions:

• Unbalanced classes break the 50 % intuition. A classifier that always answers “standard” in an 80/20 oddball scores 80 %. Use balanced accuracy, or the area under the ROC curve, or resample to balance — and say which.
• Cross-validated accuracy is itself a random variable, and its variance is not the binomial variance, because the folds share training data. Fold-to-fold standard deviations are descriptive, not a standard error you can build an interval from ( (Varoquaux et al., 2017) ).
• Small test sets make the estimate coarse. With 9 test epochs per fold, accuracy can only take ten values, and 100 % in a fold means nine correct guesses.

Permutation testing, which is the honest significance test

Shuffle the class labels, re-run the entire pipeline — cross-validation, spatial filter, classifier and all — and record the score. Repeat a few hundred or a few thousand times. The p-value is the proportion of shuffled runs scoring at or above the real one.

This is worth the compute because it is the only version that is automatically correct: it makes no distributional assumption, it accounts for the class balance and the trial count as they actually are, and — crucially — if your pipeline leaks, the permutation distribution leaks too, which sometimes exposes the leak as a null distribution centred far above chance. The exchangeability question is the same one L3.7 asked: shuffle labels within the unit that defines the null, which for a within-subject decoder means within subject and, where trials are grouped, respecting the groups.

Time-resolved decoding puts us back in L6.1

Fitting one classifier per time point gives a decoding time course — and one test per time point, which is mass univariate with a fancier statistic. Everything in L6.1 applies unchanged: the time course will wander above chance somewhere by chance alone, and the correction (cluster permutation over the time axis, or a max-statistic over the whole course) is not optional. The peak of a decoding time course is a peak of a noisy curve; its latency is no more an estimate than a cluster’s onset is (pf-cluster-inference-misread).

Temporal generalization

Train a classifier at time t and test it at every other time t’, and the result is a matrix ( (King, 2014) ). Its shape is informative about the dynamics:

• A narrow diagonal — classifiers work only at the time they were trained — indicates a sequence of transient patterns, each different from the last.
• A square block — a classifier trained at 300 ms works from 300 to 600 ms — indicates one pattern sustained over that period.
• Off-diagonal structure that reverses sign indicates a pattern that recurs with opposite polarity.

Read these as descriptions of what a linear classifier could transfer, not as claims about neural codes. The matrix is also a much larger search space than the diagonal — a full time × time grid — so it needs the same correction as everything else, over the whole matrix.

What above-chance decoding does and does not mean

It means: information distinguishing the conditions is linearly available in these sensors, under this preprocessing, at this time, to this classifier, for these trials. That is a real and useful finding, and it is narrower than the sentences usually written about it.

It does not mean:

• That the brain uses that information. Decodability is a statement about the experimenter’s access, not about downstream neural readout.
• A location. Classifier weights are not a topography. The weights of a linear model include terms whose job is to cancel noise, so a large weight can sit on a channel with no signal at all; this is precisely the filter-versus-pattern distinction of L5.7, and only the patterns are interpretable.
• A latency. See above: the peak of the curve is the peak of a noisy curve.
• An effect size comparable to anything. 86.7 % here and 86.7 % in another paper are not the same evidence unless the trial counts, the classes, the pipeline and the cross-validation match.
• That the difference is “cognitive”. Muscle activity, eye movements and electrode drift decode extremely well. A decoder finds whatever separates the conditions, including the things you were hoping to have removed — which is why a decoding paper needs its artifact handling described in more detail than an ERP paper, not less.

The two numbers this lesson is built on, with their disclosures

The cost of leakage. One subject of ds-eegbci (S002, runs R04, R08 and R12; left- against right-hand motor imagery; 8–30 Hz; 0.5–2.5 s after the cue; 45 epochs; CSP with 6 components into LDA; 5-fold stratified cross-validation with a recorded seed) scores 86.7 % with CSP fitted inside every training fold and 97.8 % with CSP fitted once on every epoch, labels included. Same subject, same epochs, same folds, same seed: +11.1 percentage points from the position of one line of code.

The subject was selected by accuracy, by a stated rule — the lowest-numbered subject whose mean cross-validated accuracy reaches 0.75. Across the twelve candidates the same pipeline runs from 44.4 % to 95.6 %, with a median of 55.6 % and a mean of 60.4 %; the shipped subject sits at 86.7 %, near the top. Quoting 86.7 % as “what motor-imagery decoding achieves” would teach exactly the selection effect this level exists to name, so the spread is part of the number. The shipped data file carries every candidate’s score for that reason, and the widget draws all twelve bars.

One more result worth keeping, because it contradicts the usual intuition: on the same subject, CSP with 2 components scores 51.1 % — worse than the 73.3 % of plain log band power at C3 and C4 with no spatial filter at all — while 4 and 6 components give 86.7 % and 8 give 88.9 %. More components is not monotonically better, and two is too few here. A pipeline’s hyperparameters are choices like any other, and choosing them by the test score is the leak this lesson opened with.

The data behind this lesson

• The decoding demonstration above uses ds-eegbci (EEG Motor Movement/Imagery Database, ODC-By 1.0, open), 64 channels at 160 Hz, no hardware filters. The numbers are properties of the shipped w-csp-explorer asset, generated by data/scripts/make_csp_products.py and recorded in data/manifest.json; the figure in the pitfall below comes from data/scripts/make_figures_p3.py over twelve subjects.
• The notebook works on ds-erpcore P3 instead, because time-resolved decoding of a target-versus-standard contrast is the case the objectives describe and the paradigm is the one Levels 2 and 3 already built a pipeline for.
• TODO(confirm): ERP CORE’s licence is contested at source; derived assets ship under the strictest reading, CC BY-SA 4.0, per the author’s decision of 2026-09-18 (§13 item 15).

Explore

This lesson has no widget of its own, but the numbers it turns on are on screen in the CSP explorer lab, which Level 5 introduced. Open it in its pipeline mode and read three panels in this order:

• the subject survey: twelve bars, one per candidate, with the shipped subject marked. Look at the spread before you look at the winner, and notice that the chance line sits inside several of the bars’ uncertainty;
• the leakage demonstration: the same subject, the same epochs, the same folds and the same seed, with the spatial filter fitted inside the fold and then once on everything. The gap is the exercise below;
• the stage list: no spatial filter, then CSP with 2, 4, 6 and 8 components. Two components score below no spatial filter at all, which is worth sitting with before you tune a component count by its score.

Then, on paper, take a decoding pipeline you have run or read about and mark every step as label-aware or not. That list, not the classifier, is where the result is decided.

Practice

Decoding as inference: time-resolved decoding of the P3 with a permutation test, chance and its variance, temporal generalization, and leakage measured three ways nb-6-5-decoding

Level 6 ~12 min

notebooks/L6/nb-6-5-decoding.ipynb

Downloads from ds-erpcore, ds-eegbci.

Open in Colab Download Read it here

SlidingEstimator over the ds-erpcore P3 epochs with a scaler and a logistic regression inside a pipeline, cross-validated so that every fitted step stays inside the fold, scored with the area under the ROC curve because the classes are unbalanced by design. Then the permutation test — the whole pipeline re-run on shuffled labels — and a cluster-based correction over the time axis, so the significance statement covers the whole time course rather than the point that happened to be highest. A final section builds the temporal generalization matrix and states what its shape does and does not license.

Exercises

Exercise ex-6-5-peak-decoding-time

Numeric

From the notebook's time-resolved decoding of target versus standard: at what latency, in milliseconds after stimulus onset, does the decoding time course peak?

Your answer ms

No answer key yet — work it out and compare with the notebook.

Check

Hint

Exercise ex-6-5-permutation-significance

Multiple choice

The notebook shuffles the class labels 1,000 times, re-runs the whole cross-validated pipeline each time, and compares the observed time course against the resulting null. What does the p-value at the peak, corrected over the time axis by the cluster procedure, license you to say?

Options

That decoding is above chance at that specific latency, and therefore that the information becomes available then.That the conditions are distinguishable from these sensors somewhere in the tested time window, with the error rate controlled over the whole window.That the classifier would achieve the same accuracy on a new participant.That the observed accuracy is significantly higher than 50 %, which is what the permutation distribution is centred on.

Check

Hint

Exercise ex-6-5-leakage-cost

Numeric

One subject, 45 epochs, one pipeline, one seed. With the spatial filter fitted inside every training fold the cross-validated accuracy is 86.7 %; with the same filter fitted once on all the epochs, labels included, it is 97.8 %. How many percentage points does the leak buy?

Your answer percentage points

Accepted within ±0.2 percentage points.

Check

Hint

Exercise ex-6-5-inside-the-fold

Multiple select

You are cross-validating a decoder. Select every step that must be fitted inside each training fold rather than once on the whole dataset.

Options (select all that apply)

A CSP or xDAWN spatial filter.A fixed 8–30 Hz band-pass applied to the continuous data before epoching.Selecting the 20 channels with the largest condition difference.Standardizing each feature to zero mean and unit variance.Choosing the classifier’s regularization strength by comparing scores.Rejecting epochs whose peak-to-peak amplitude exceeds a fixed threshold on any channel.Resampling the training set to balance the two classes.

Check

Hint

Pitfalls

Pitfall

Information leakage across CV folds

Symptom

Cross-validated accuracy collapses on a true held-out set.

Cause

Cross-validation estimates performance on unseen data. The estimate is valid exactly to the extent that the held-out fold is unseen by the whole pipeline, not just by the classifier. Any step fitted on all the data before the split has already looked at the test fold, so the classifier is scored on data it has partly been fitted to.

Detect

• Move the step inside the fold and re-run. The difference is the leak, measured in the units the paper reports. This is the only test that gives a number, and it takes minutes. - Permutation-test the entire pipeline. Shuffle the labels, re-run cross-validation, spatial filter and classifier together, a few hundred times. If the resulting null distribution is centred above chance, something in th…

Fix

• Make the pipeline one object and fit it per fold. A Pipeline containing the spatial filter, the scaler and the classifier, passed to the cross-validator, leaks nothing by construction — the fix is structural rather than vigilant, which is what makes it reliable. - Nest the hyperparameter search. An inner cross-validation on the training folds chooses the parameters; the outer fold scores them.…

Full entry with example →

In other tools

In other toolsFieldTrip — names only

The equivalents of what this lesson does, for a reader who works in another toolbox. Function names only: their own documentation is the place to learn how to call them.

FieldTrip

• ft_statistics_crossvalidateFieldTrip
• ft_timelockstatistics(mvpa)FieldTrip

Names checked 2026-09-18 against FieldTrip 20251218.

Reading

1. King & Dehaene (2014). Temporal generalization. unverified
2. Varoquaux et al. (2017). Assessing brain decoders. unverified