Information leakage across CV folds

Symptom

A decoding accuracy that is much higher than the paradigm’s literature usually reports, and that does not survive contact with genuinely new data: the classifier scores in the nineties in cross-validation and near chance on a held-out session, a held-out run, or the next participant. Other signs, any one of which is worth a second look:

• Fold-to-fold variance is implausibly small for the number of test trials. With nine test epochs per fold, accuracy can only take ten values; a tight spread across folds usually means the folds are not independent.
• A permutation null centred above chance. Shuffle the labels, re-run the whole pipeline, and the null distribution sits at 65 % rather than at 50 %.
• Accuracy that barely moves when the classifier is weakened — fewer features, stronger regularization, a simpler model — because the information is not coming from the classifier.
• Near-ceiling accuracy on a few dozen trials, in a paradigm where the published range is 55–75 %.
• A methods section whose pipeline order cannot be reconstructed, so that where the spatial filter or the feature selection was fitted is simply not stated.

The characteristic history is that nobody notices until someone else tries to reproduce it, because within the original analysis the number is stable, reproducible and wrong.

Cause

Cross-validation estimates performance on unseen data. The estimate is valid exactly to the extent that the held-out fold is unseen by the whole pipeline, not just by the classifier. Any step fitted on all the data before the split has already looked at the test fold, so the classifier is scored on data it has partly been fitted to.

The steps that leak, in rough order of how much damage they do:

• Supervised spatial filters. CSP, xDAWN and any other projection estimated from labelled data. These have many parameters and are fitted on the covariance structure of both classes, so fitting them once on every epoch is the largest single leak available in EEG decoding (L5.7).
• Feature, channel or time-window selection performed by comparing the conditions. Selecting “the 20 channels with the biggest difference” on all the data is the decoding form of pf-post-hoc-windows.
• Hyperparameter choice by score. Picking the regularization strength, the number of components, the frequency band or the epoch window because it gave the best cross-validated result makes that result an optimum over a search, not an estimate. This needs an inner loop — nested cross-validation.
• Class balancing or resampling applied before the split, which moves label information across the boundary.
• Standardization fitted on all the data. No labels are involved, so the leak is smaller, but the test fold still contributes to the training representation.
• Artifact rejection tuned on the outcome — a threshold lowered until the accuracy improved.

Two structural leaks are independent of which steps you fit where, and they are the ones that survive an otherwise careful pipeline:

• Trials that are not independent. Epochs adjacent in time, epochs from the same block or run, and overlapping epochs share drift, artifact and slow state. A random k-fold split puts near-duplicates on both sides of the boundary, and the classifier recognises the segment rather than the condition. Where the data have run, block or session structure, the split has to respect it.
• The wrong held-out unit for the claim. Within-subject cross-validation holds out trials, so it estimates generalization to new trials of that participant. A claim about people needs the participant to be the held-out unit, which is a different and usually much harder problem.

The severity scales with the number of parameters the leaked step carries and inversely with the number of trials. On small EEG datasets — a few dozen epochs per subject — it is very large.

Detect

• Move the step inside the fold and re-run. The difference is the leak, measured in the units the paper reports. This is the only test that gives a number, and it takes minutes.
• Permutation-test the entire pipeline. Shuffle the labels, re-run cross-validation, spatial filter and classifier together, a few hundred times. If the resulting null distribution is centred above chance, something in the pipeline is learning from the labels outside the fold. A null centred at chance is not proof of correctness, but a null above chance is proof of leakage.
• Check the split against the data’s own structure. List what each epoch belongs to — run, block, session, participant — and confirm that no group appears on both sides of any fold boundary. Group-aware splitters exist for this.
• Hold out something genuinely untouched, once, at the end: a whole session, a whole run, or a set of participants that no decision in the analysis has seen. Report that number beside the cross-validated one.
• Read the code, not the methods. The leak is the position of one line. A methods section can describe a correct pipeline that the code does not implement, and neither the author nor the reviewer will notice from the prose.
• Compare against the paradigm’s published range. An accuracy well above what the literature reports for the same task, the same electrode count and the same trial count is a hypothesis about your pipeline before it is a finding about the brain.

Fix

• Make the pipeline one object and fit it per fold. A Pipeline containing the spatial filter, the scaler and the classifier, passed to the cross-validator, leaks nothing by construction — the fix is structural rather than vigilant, which is what makes it reliable.
• Nest the hyperparameter search. An inner cross-validation on the training folds chooses the parameters; the outer fold scores them. Report both loops.
• Split on the group, not the trial. Where runs, blocks or sessions exist, hold them out whole. Where the claim is about people, hold out people.
• Keep the genuinely fixed steps outside and say that they are fixed. A band-pass, an epoch window or a rejection threshold decided in advance and applied identically to every epoch sees no labels and can sit outside the loop. The qualifier is doing the work: a band chosen because it decoded best is label-aware by the back door.
• Report chance with its interval, not as a number. On 45 epochs the 95 % binomial band around 50 % runs from 35.4 % to 64.6 %, so a single subject at 62 % is inside it.
• Report the held-out number separately from the cross-validated one, and say which decisions were made before each.
• Pre-register the pipeline, including where every step sits relative to the fold boundary. The leak is hardest to introduce when the position was written down before the score existed.
• Publish the code. Leakage is a property of an implementation, and it is the class of error that peer review of prose cannot catch.

Example

12 EEGMMIDB subjects, motor-imagery runs R04/R08/R12, 8–30 Hz, 0.5–2.5 s after the cue, CSP with 6 components into LDA, 5-fold stratified cross-validation with one seed. The only difference between the two bars is where CSP is fitted. Fitting it once on every epoch — labels included — before cross-validating the classifier raises the median accuracy from 0.556 to 0.967, a median inflation of +0.378, and it does so for every subject here. The filters have already seen the test folds’ labels, so the classifier is being scored on data it has partly been fitted to. Any step that looks at labels — spatial filtering, feature selection, channel or window choice, normalisation — belongs inside the fold. (Figure and caption generated by data/scripts/make_figures_p3.py from ds-eegbci, ODC-By 1.0, dataset DOI 10.13026/C28G6P; derived: filtered, epoched, spatially filtered, cross-validated. The sidecar’s own alt and caption are the authority.)

Two notes that belong with the number rather than after it.

The single-subject version is smaller and is the one L6.5 quotes. Subject S002 of the same sweep scores 86.7 % honest against 97.8 % leaked — the same subject, the same 45 epochs, the same folds, the same seed, and +11.1 percentage points for the position of one line. The figure’s median inflation of +37.8 points is larger because the median subject’s honest accuracy is far lower, so there is more room between it and the ceiling; both numbers come from the same pipeline and neither contradicts the other.

The panel also shows why one subject is not the story. Honest accuracy across the twelve runs from 44.4 % to 95.6 %, with a median of 55.6 %. The 86.7 % subject that the w-csp-explorer asset ships was chosen by a stated rule — the lowest-numbered subject reaching 0.75 — which is itself a selection effect, and the shipped file carries every candidate’s score so that the selection stays visible. TODO(confirm): site/notes/widgets-L.md and the Phase 3 addendum record that median as 60.0 %, which is the seventh of the twelve sorted per-subject means rather than the average of the sixth and seventh; the shipped w-csp-explorer/csp.json survey and this figure’s own sidecar both give 55.6 %. The range and the shipped subject’s 86.7 % agree in all three. Recorded rather than silently corrected.