Statistics for ERPs

Why this matters

There are two honest ways to test an ERP effect and one very common dishonest one. You can measure the component in a window fixed in advance and run a single paired test; or you can test the whole waveform at once with a procedure that controls the error rate over everything it searched. What you cannot do is test every time point, plot the significant ones and describe the result — and the cluster-based permutation test, which is the standard answer to that problem, is itself routinely over-read. This lesson covers the machinery and then, at equal length, what the machinery entitles you to say.

Concepts

The simple case: a test on a measurement

If L3.3 did its job, you have one number per subject per condition: the mean amplitude at stated electrodes in a window fixed before the data were seen. Then the test is the ordinary one — a paired t-test across subjects on the condition difference, or a repeated-measures ANOVA or linear mixed model when the design has more factors (L6.2).

Three points that are easy to lose:

• The unit of analysis is the subject. Trials are nested inside subjects; testing across trials while ignoring the subject grouping treats non-independent observations as independent and inflates the error rate badly (L6.2, and the pseudoreplication argument in L6.3).
• One test per pre-specified measurement. If you measured three components at four electrode clusters, you ran twelve tests, and the family has to be accounted for.
• Report the effect size and its interval, not only the p-value (L6.3). An ERP amplitude difference in µV with a confidence interval is a more useful result than a p, and it is what the next study will need in order to plan its own trial counts.

This route is cheap, powerful when the window is right, and blind everywhere the window is not. Its weakness is exactly the pre-specification that makes it valid: if the component sits elsewhere in your sample, you will not find it, and moving the window afterwards is pf-post-hoc-windows.

Mass-univariate testing, and why uncorrected is wrong

The alternative is to give up the single window and test everywhere: one statistic per channel and per time point. A 30-channel epoch sampled at 250 Hz over one second is 7,500 tests.

Run them at α = .05 and, when nothing at all is happening, about 5% of them — roughly 375 points — will come out “significant” anyway. They will be scattered across the map in small blobs, because neighbouring samples and neighbouring channels are correlated, so the false positives cluster too and look structured. This is pf-uncorrected-timepoint-tests: the display is not a set of findings, it is a picture of the α level.

The classical corrections are available and are usually unattractive here. Bonferroni over 7,500 correlated tests is so conservative that only enormous effects survive. False discovery rate controls a different and weaker quantity — the expected proportion of false positives among the rejected tests — which is a defensible thing to control but is not what most ERP claims assert. The reason both are unattractive is the same: they treat the tests as an unstructured family, when in fact an EEG effect is expected to be contiguous — spread over neighbouring time points and neighbouring electrodes. A procedure that exploits that structure can be far more powerful at the same error rate.

Permutation: the general idea

A permutation test builds the null distribution from the data rather than assuming one. The recipe:

1. Compute the statistic you care about on the real data.
2. Repeatedly relabel the data in a way that would make no difference if the null were true, recomputing the statistic each time.
3. The p-value is the proportion of relabelled statistics at least as extreme as the real one.

For a within-subject ERP contrast, the exchangeability assumption is specific: under the null, the two conditions are interchangeable within each subject, so flipping the sign of a subject’s difference wave leaves the distribution unchanged. The permutation is therefore a random sign flip per subject, and with n subjects there are 2ⁿ distinct sign patterns — so the null is exact (up to Monte Carlo error) rather than assumed.

Everything depends on getting the exchangeability right. Sign-flipping subject-level difference waves is the correct scheme for a paired design; it is not the correct scheme for a between-group comparison, where the permutation shuffles group labels instead. Getting this wrong is the standard way a permutation test becomes invalid.

The cluster statistic and the max-statistic null

(Maris, 2007) combines the permutation idea with the contiguity structure:

1. Compute a t statistic at every channel and time point.
2. Keep points whose statistic exceeds a cluster-forming threshold, and group adjacent surviving points — adjacent in time and in the channel neighbourhood graph — into clusters.
3. Summarise each cluster with one number, conventionally the sum of the t values inside it.
4. For each permutation, repeat steps 1–3 and record only the largest cluster statistic anywhere in the map.
5. Compare the observed clusters against that distribution of maxima.

Step 4 is the whole trick. Because the null is a distribution of maxima over the entire search space, a cluster that beats it has beaten everything the analysis looked at, and the family-wise error rate over all channels and time points is controlled at α by construction. There is no further correction to apply, and — the point that matters for the next section — no individual point in the map receives a p-value of its own.

The cluster-forming threshold is a separate choice from α, and it does not change the error rate. Lowering it merges points into fewer, larger clusters with bigger t-sums; but every permutation gets bigger clusters too, so the null grows with them. What the threshold changes is sensitivity to different shapes of effect: a high threshold favours strong, focal effects, a low one favours weak, extended effects. It must be fixed in advance, because sliding it until a cluster appears is a search like any other.

Threshold-free cluster enhancement (TFCE) removes the choice by integrating cluster support over all thresholds; it is treated in L6.1 with the rest of the multiple-comparisons landscape.

What a cluster p-value licenses — precisely

This is the part of the lesson that most changes how people write. A significant cluster licenses:

• That the conditions differ somewhere in the tested space. The null hypothesis being rejected is the exchangeability of the condition labels within subject, over the whole set of channels and time points that were tested. That is one hypothesis and one test.
• That the family-wise error rate over the entire search is α. This is a corrected result; it needs no further adjustment.
• A description of this sample: at the chosen threshold, the suprathreshold points formed a blob covering these channels between these two latencies, with its largest t here. Reported as a description — “the cluster extended over…” — and not as an estimate.

It does not license:

• An onset. “The effect began at 340 ms” is the canonical error (pf-cluster-inference-misread). The cluster’s first time point is where the point-wise statistic happened to cross the cluster-forming threshold in this sample; move the threshold and the onset moves with it. The test provides no confidence interval for onset, and (Sassenhagen, 2019) shows that the cluster boundary is a biased estimate of it.
• An offset or a duration. The same argument at the other edge.
• A location. “The effect is parietal” or “is maximal at Pz” attributes to the test a resolution it does not have. Points inside the cluster were not individually tested; points outside it were not shown to be null.
• Point-wise significance. “812 of the 7,500 points were significant” counts suprathreshold points, which is the uncorrected error wearing corrected clothing.
• A comparison between clusters. A cluster p-value is not an effect size. It depends on extent, on the threshold and on the noise, so “cluster 1 is stronger than cluster 2” does not follow from its being smaller.
• Absence of an effect. A non-significant cluster, or no cluster at all, is not evidence that the conditions do not differ.

Reporting a cluster test

What a reader needs in order to reproduce and to judge: the contrast and the unit of analysis; the tested space — which channels, which time window, at which sampling rate; the point-wise statistic; the cluster-forming threshold and how it was chosen; the neighbour definition for channel adjacency; the permutation scheme (sign-flip for a within-subject design) and the number of permutations; the random seed; α and whether the test was one- or two-tailed; and, for each surviving cluster, the p-value, the cluster statistic, the extent described as an extent, and a figure showing the map with the cluster outlined. Report the non-significant clusters too, or say that none formed.

The data behind this lesson

• ds-erpcore P3, CC BY 4.0, open access, per-subject downloadable; 30 EEG + 3 EOG channels in a 10-20 placement scheme, 1024 Hz, CMS reference, 60 Hz mains, no software filters, 40 participants per paradigm. TODO(confirm): the author mirrors the ERP CORE entry into the catalogue registry and signs off the dataset page (§10.11 item 8); the shipped asset sidecars also record a licence conflict in the source — the OSF node record says CC BY 4.0, the per-paradigm component’s own LICENSE file says CC BY-SA 4.0 and its dataset_description.json says CC0 — which the author reconciles (§13 item 22).
• The widget serves a precomputed observed t map over channels and time for the target-minus-standard contrast, the candidate clusters with their own p-values, a stored set of permutation frames that drive the animation, and the full null distribution of maximum cluster statistics that drives the p-value. As shipped: 19 subjects, 30 channels, 129 time points from −0.2 to 0.8 s, a cluster-forming threshold set at the two-tailed .05 point of the t distribution on 18 degrees of freedom, 1,024 sign-flip permutations, a recorded seed, and channel adjacency built from the montage by Delaunay triangulation. One subject was excluded for having too few surviving trials, with the criterion and the count recorded. Read the frame’s provenance line and the shipped data file for what actually shipped.
• One number in that file is worth knowing before you read any cluster p-value: because the identity relabelling counts as one of the 1,024 permutations, the smallest attainable p is 1/1024 ≈ .001. A p at that floor means “smaller than this null’s resolution”, not “essentially zero” — and adding permutations is the only thing that lowers the floor.

Explore

Work it in this order: turn on the uncorrected map first and count the scattered specks against the number expected by chance; step the permutation player one frame at a time and watch each relabelled map produce its own largest cluster; let it run and watch the histogram of maxima build, with the observed cluster’s position in it; move the cluster-forming threshold and confirm that the clusters change shape while the p-value does not collapse, because the null moved too; then read the “what this licenses” panel for the selected cluster and check its second list against the sentence you would have written.

Practice

The notebook runs permutation_cluster_1samp_test on subject-level ds-erpcore difference waves, with the neighbour definition, the threshold, the number of permutations and the seed all stated in one cell, and prints the cluster table. Its final cell prints the statistics the exercise refers to.

Exercises