The multiple-comparisons landscape

Objectives

• Enumerate the search space (channels × time × frequency × conditions)
• Compare FWER vs FDR, Bonferroni, max-statistic, cluster permutation and TFCE
• Pre-specify ROIs and windows

Why this matters

Two numbers state the whole of Level 6. Take twenty subjects of a real oddball experiment, destroy the effect by shuffling each subject’s condition labels so that the null is true by construction, and run 486 analysis pipelines that a reviewer would accept without comment. At least one of them reaches p < .05 in roughly three simulated datasets out of four. A single pipeline fixed in advance reaches it in about one in twenty. The second number is the calibration check — under a true null a fixed analysis should be wrong about α of the time, and it is.

Both are Monte-Carlo estimates and are quoted here to the precision they support, which in a lesson about overstated precision is not a stylistic choice. The shipped 200-draw sweep gives 79.5 % and 4.0 %; rerunning the identical construction at 2,000 draws gives 75.4 % and 5.25 %. The 200-draw figure carries a 95 % interval of 74–85 %, so the two runs agree — and neither supports three significant figures. The widget prints its own sweep’s numbers, and the exercise below asks you to read them; the claim this lesson makes is the one those numbers can carry. The gap between the two is not a gap between honest and dishonest analysts. It is a gap between choosing before and choosing after.

This lesson is about the half of that gap a statistical correction can close: the search over channels, time points, frequencies and conditions that one analysis performs in a single pass. L6.4 is about the other half, which no correction touches.

Concepts

The search space is larger than it looks, and it is easy to count

Start with the part that is pure arithmetic. A 64-channel recording, epoched and resampled so that the analysis window holds 200 time points, tested one channel and one time point at a time, is

• 64 channels × 200 time points = 12,800 tests,
• and at α = .05, under a null that is true everywhere, 12,800 × 0.05 = 640 of them are expected to come out significant.

Six hundred and forty “significant” points in a recording where nothing happened. They will not be scattered like confetti either: neighbouring samples and neighbouring electrodes are correlated, so the false positives arrive in smooth blobs that look exactly like components. That is pf-uncorrected-timepoint-tests, and it is the same arithmetic L3.7 ran on a smaller grid — 30 channels × 250 samples, 7,500 tests, about 375 expected.

Now add the dimensions Level 4 introduced. A time-frequency analysis multiplies the grid by the number of frequency bins; a design with more than two conditions multiplies it by the number of contrasts; a connectivity analysis over sensor pairs squares the spatial term. The count grows fast, and every one of these is a measurement dimension: a coordinate of the data, fixed before the analysis started, and countable.

Which error rate is being controlled

“Correcting for multiple comparisons” is not one thing. Three quantities are in use, and they answer different questions:

• Per-comparison error rate — the probability that this test is a false positive. This is what an uncorrected α controls, and over 12,800 tests it is not the quantity anyone means.
• Family-wise error rate (FWER) — the probability of one or more false positives anywhere in the family. Controlling FWER at .05 means that, under a complete null, 95 % of studies report nothing at all. This is what most EEG claims implicitly assert.
• False discovery rate (FDR) — the expected proportion of false positives among the tests you rejected. Controlling FDR at q = .05 means that, on average, one in twenty of the points you called significant is not. It is a weaker and often more useful guarantee, and it is a different claim: an FDR-controlled map is a map with an expected error budget, not a map you can point at one pixel of.

Neither is more correct than the other. What is incorrect is reporting one and meaning the other.

The procedures, and what each buys

Procedure	Controls	Assumes	Strength	Cost
Bonferroni (α/m)	FWER	Nothing — it holds under any dependence	Trivial to compute and to describe	Ruinously conservative over thousands of correlated tests
Holm (step-down)	FWER	Nothing	Uniformly more powerful than Bonferroni, same guarantee	Still treats the family as unstructured
Benjamini–Hochberg	FDR	Independence or positive dependence	Far more sensitive on large maps	Controls a proportion, not an occurrence; no point-wise claim
Max-statistic permutation	FWER	Exchangeability under the null	Exact, uses the real dependence in the data	Only as sensitive as the largest single statistic
Cluster permutation	FWER	Exchangeability; effects are contiguous	Powerful for extended effects	The cluster is not localized or dated (pf-cluster-inference-misread)
TFCE	FWER	Exchangeability	No cluster-forming threshold to choose	Two shape parameters instead; slower

Three observations about that table are worth more than the table.

Bonferroni is not wrong, it is mismatched. Dividing α by 12,800 gives a per-test threshold near .0000039. Nothing in an ordinary ERP experiment survives it. The procedure is valid; it just spends the entire error budget on the assumption that the tests could be adversarially dependent, when in fact they are smoothly and positively dependent in a way you know about in advance.

Permutation exploits exactly that structure. The max-statistic test computes the statistic everywhere, keeps only its maximum over the whole map, and repeats under relabelling. A point that beats the distribution of maxima has beaten everything the analysis looked at, so the FWER over the entire search is α by construction, with no further correction — and the null was built from your own data’s correlation structure rather than assumed. The price is that the test is driven by one number, so a broad, shallow effect can lose to the noise’s sharpest spike.

Cluster permutation is the same machinery with a different statistic. Instead of the maximum point, it takes the maximum cluster — a contiguous run of suprathreshold points, summarised by its t-sum — which matches the shape an EEG effect actually has. That is why it dominates the literature, and why L3.7 spends half its length on what the resulting p-value does not license.

TFCE: removing the threshold, not the choice

The cluster-forming threshold is an analysis choice that changes which effects the test can see: high thresholds favour focal, strong effects; low thresholds favour extended, weak ones. It does not change the error rate — the null moves with it — but it does change the answer, and sliding it until a cluster appears is a search like any other.

Threshold-free cluster enhancement removes the choice by refusing to make it. At each point it integrates the support that point receives from clusters at every threshold, weighting each cluster’s extent and height, and produces a new statistic image; that image is then submitted to the same max-statistic permutation. The result is a point-wise, FWER-corrected map with no threshold to declare ( (Smith, 2009) ).

What TFCE does not remove is judgement. The integration is weighted by two exponents, conventionally fixed at the values the original work proposed, and those are parameters like any other — defensible because they are conventional and fixed in advance, not because they are absent. (Pernet et al., 2015) compares these procedures on EEG-shaped data, and the useful finding there is not that one wins, but that they differ in which effects they find, which is why the choice belongs in a pre-registration rather than in a results section.

Shrinking the space before you test it

Every correction above is a way of paying for a large search. The other move is to search less.

• A pre-specified region and window. One measurement — mean amplitude at Pz over 300–600 ms, say — is one test. It needs no correction at all, it is the most powerful analysis available when the window is right, and it is blind everywhere the window is wrong. The condition is that the window came from somewhere other than the data being tested: prior literature, your own earlier sample, or a pre-registration.
• A collapsed localizer. When the component’s latency in your sample is genuinely uncertain, choose the window and electrodes from the average of all conditions together. Under the null that average carries no information about the condition difference, so the selection stays orthogonal to the test (L3.3, and pf-post-hoc-windows).
• A declared family. If you measure three components at four electrode clusters, you ran twelve tests. Say so, and correct over twelve — a number small enough that Bonferroni is perfectly reasonable.
• A hierarchy. One pre-specified confirmatory test, reported as confirmatory; a mass-univariate corrected analysis of the rest, reported as exploratory. Both are legitimate; mixing them after the fact is not.

The second search space, which no correction touches

Every procedure above corrects for a search over the data. None of them corrects for a search over analyses.

The widget below runs 486 pipelines over the same twenty subjects: three references, three high-pass cutoffs, two baseline intervals, three rejection criteria, three a-priori measurement windows, three electrode choices. Each pipeline applies one correction-free test to one pre-specified measurement, and each is individually defensible — the data file states the published precedent for every option. On data whose null is true by construction, the pre-specified pipeline rejects in about one draw in twenty and at least one of the 486 rejects in about three draws in four.

Note what that comparison is not. It is not a claim that anyone ran 486 analyses. (Gelman, 2014) ’s point is that the inflation arrives even when only one analysis is ever run, provided the choice of that analysis could have gone differently had the data looked different — and a competent analyst making one defensible decision at a time is exactly the person that rate describes. The fix is not vigilance. It is fixing the choices in advance, or reporting the whole grid; L6.4 does both.

The data behind this lesson

• ds-erpcore P3 (active visual oddball), 20 subjects, target-minus-standard mean amplitude. The shipped grids hold 486 outcomes each, with the choice set, the rationale for every option, the parameters held fixed, and the construction of the null. Epochs −0.2 to 0.8 s at 256 Hz (from 1024 Hz), 30 Hz low-pass, ocular correction by regression, two-sided paired t-test across subjects. Generated by data/scripts/make_forking_paths.py; the frame’s provenance line is the authority.
• The null construction, stated because it is what makes the rate meaningful: within each subject the vector of condition labels is permuted, leaving the epochs, their order, their artifacts and the 40/160 target/standard counts untouched. The expected difference is therefore exactly zero at every channel and latency, so every p < α in that grid is a false positive. Within one draw all 486 paths share the permutation — one dataset, many analyses, which is the situation an analyst is actually in. Across the 200 draws the permutations are independent, which is what makes the result a rate rather than an anecdote — and what lets it be quoted with an interval rather than a decimal point.
• The cluster widget serves the same paradigm tested properly: 19 subjects, 30 channels, 129 time points, 1,024 sign-flip permutations, the full null of maximum cluster statistics, and every candidate cluster including the ones that did not survive.
• TODO(confirm): ERP CORE’s licence is contested at source — the shipped LICENSE file says CC BY-SA 4.0, dataset_description.json says CC0, and the OSF node record says CC BY 4.0. The site ships derived assets under the strictest reading, CC BY-SA 4.0, per the author’s decision of 2026-09-18 (§13 item 15); the author still reconciles the three statements and mirrors the entry into the catalogue registry.

Explore

Garden of forking paths, null mode — 486 defensible pipelines on data with nothing to find

mode: null Open lab page →

Reset Share state Fullscreen

Loading Garden of forking paths, null mode — 486 defensible pipelines on data with nothing to find…

What to look for

• Read the two counts side by side: how many of these paths reached p < α in this dataset, and how many would be expected if the same number of independent tests were run on a true null. Both are printed; neither is a judgement about anyone.
• Every path here is one a methods section could state without apology. Walk a few: change the high-pass, change the reference, move the window. The answer moves. Nothing about the data changed — only the description of how it was analysed.
• Look at the whole distribution, not the path you happen to be standing on. A single p-value is one draw from the spread you are looking at, and the spread is what the dataset actually supports.
• Draw another dataset. In most draws nothing at all reaches α; in a few, dozens of paths do. That skew is the correlation between paths: they share the same data, so they are far from independent, and "a few false positives here and there" is not how it arrives.
• Pick one path before you look, and stick to it across several draws. That single pre-specified pipeline is wrong about α of the time — which is exactly what α promises, and exactly what stops being true the moment the choice is made after seeing the result.

Data provenance is recorded in each asset's sidecar under /data/widgets/w-garden-of-forking-paths/.

What to look for:

• The two rates, side by side. The panel prints how often at least one of the 486 paths reached α across the producer’s 200-draw sweep, and how often the single pre-specified path did. The second is the calibration check: a fixed analysis of a true null must be wrong at about α, and 4.0 % says the construction is sound. The first is the cost of choosing afterwards.
• The yardstick, and why it is only a yardstick. Beside the count of significant paths in the current draw sits 486 × α = 24.3, the number you would expect from that many independent tests. The paths are not independent — they share the data — so the observed counts swing far above and far below it.
• Draw another dataset, several times. Most draws produce nothing at all; a few produce dozens; the worst in the sweep produced 220 of 486. False positives from a forking analysis do not arrive as a steady drizzle, they arrive in floods, and a single quiet draw is not reassurance.
• Walk a path deliberately. Change only the reference, or only the high-pass, and watch the p-value move while the data stand still. Read the rationale printed under each control: these are not straw men, they are the options a methods section states without apology.
• Fix one path before you look, and keep it across draws. That is the pre-specified pipeline, and it behaves exactly as advertised. Nothing about the analyst changed between the two numbers — only the order of looking and choosing.

Cluster permutation, ERP mode — the corrected comparison on the same paradigm

mode: erp Open lab page →

Reset Share state Fullscreen

Loading Cluster permutation, ERP mode — the corrected comparison on the same paradigm…

What to look for

• Turn the uncorrected map on: every channel × time point with |t| above the α critical value lights up, and specks appear all over the map. At α = .05 roughly 5 % of the points light up even where nothing is happening — that is the expected number, not a finding (pf-uncorrected-timepoint-tests).
• Step the permutation player. Each permutation is a relabelling under the null; its largest cluster is nearly always smaller than the observed one, and the histogram of those maxima is the null distribution the observed cluster is compared against.
• Watch the p-value while the player runs: it is simply the proportion of the null at or above the observed statistic, so it is coarse early on (one permutation in ten) and settles as the null fills. The value quoted in a paper comes from the full null, not from the first few frames.
• Lower the cluster-forming threshold: clusters merge into fewer, larger ones and their t-sums grow — but so does the null, because every permutation gets larger clusters too. The threshold changes what counts as a cluster, not how strict the test is; the family-wise error rate stays at α either way.
• Read the "What this licenses" panel for the selected cluster. A cluster p-value licenses "the conditions differ somewhere in the tested channels and time window"; it does not license an onset, an offset, a peak channel or a duration (pf-cluster-inference-misread).

Data: ds-erpcore , subject 19 subjects (sub-001..sub-020), run task-P3, -0.19921875–0.80078125 s · license CC-BY-SA-4.0 · DOI 10.18112/openneuro.ds003069.v1.0.0 · labels: algorithmic · a cropped, re-referenced or filtered derivative of the source recording. Share-alike. This asset is derived from a source whose licence requires that anything built from it carry the same licence. If you reuse it, distribute your version under CC-BY-SA-4.0 and keep the attribution below.
Kappenman, E., Farrens, J., Zhang, W., Stewart, A. X., & Luck, S. J. (2020). ERP CORE: An Open Resource for Human Event-related Potential Research. PsyArXiv. Paper DOI 10.31234/osf.io/4azqm. Dataset DOI 10.18112/openneuro.ds003069.v1.0.0, https://osf.io/thsqg/. Licensed CC-BY-SA-4.0; this is a derived asset and is distributed under the same licence (data/directory.yaml license_decision, section 13 item 15).

What to look for:

• Turn on the uncorrected map first and count the scattered specks against the number the arithmetic above predicts. That picture is the α level, not a finding.
• Step the permutation player and watch each relabelled map contribute its single largest cluster statistic to the histogram. The null is a distribution of maxima over the whole search; that is what makes the resulting p-value already corrected.
• Move the cluster-forming threshold and watch the clusters change shape while the p-value does not collapse — because every permutation’s clusters changed too.
• Read the licensing panel and compare it with the claim you would have written. The test says “the conditions differ somewhere in the tested space, with family-wise error controlled”; it does not say when, where, or for how long.

Practice

The multiple-comparisons landscape: one ERP contrast through six inference strategies, their error rates measured on a true null, and the family-wise error rate of the search itself nb-6-1-corrections

Level 6 ~8 min

notebooks/L6/nb-6-1-corrections.ipynb

Downloads from ds-erpcore.

Open in Colab Download Read it here

One contrast, five correction strategies, on the same map: uncorrected, Bonferroni, Benjamini–Hochberg FDR, max-statistic permutation and cluster permutation, with TFCE alongside where the implementation allows. The comparison to watch is not which one “found more” but which claim each result supports, and how the number of surviving points changes when the tested space is narrowed to a pre-specified region before any of them runs.

Exercises

Exercise ex-6-1-search-space

Numeric

You test 64 channels at every one of 200 time points, one t-test per channel and time point, at α = .05, on data where the null is true everywhere. How many tests is that, and how many do you expect to come out significant?

Number of uncorrected tests tests

Expected significant tests under the null tests

Exact answer required, in tests.

Check

Hint

Exercise ex-6-1-forking-rate

Numeric

From the garden widget in null mode: across the 200 simulated datasets in the shipped sweep, in what percentage of them did at least one of the 486 paths reach p < .05, and in what percentage did the single pre-specified path reach it?

At least one path significant (% of draws) % of draws

Pre-specified path significant (% of draws) % of draws

Accepted within ±0.5 / 0.5 % of draws.

Check

Hint

Exercise ex-6-1-what-corrections-control

Multiple select

Select every statement that is true.

Options (select all that apply)

Bonferroni at α = .05 over 12,800 tests controls the probability of one or more false positives at .05, whatever the dependence between the tests.Benjamini–Hochberg at q = .05 is designed to control the expected proportion of false positives among the tests it rejects, not the probability that any false positive occurs.A max-statistic permutation test controls the family-wise error rate over the whole tested space without assuming the tests are independent.Each time point inside a significant cluster receives a corrected p-value of its own from the cluster test.TFCE removes the cluster-forming threshold as a free choice by integrating cluster support over all thresholds.Restricting the analysis to one pre-specified electrode and window is valid only if the restriction did not come from the data being tested.A correction for multiple comparisons also accounts for the number of different pipelines the analyst could have run.

Check

Hint

Pitfalls

Pitfall

Mass univariate tests without correction

Symptom

Significant blobs everywhere.

Cause

Testing every channel and every time point at α = .05 means running one test per point and accepting a five per cent false-positive rate on each. A 30-channel epoch sampled at 250 Hz over one second is 7,500 tests, so about 375 points are expected to pass by chance when nothing at all is happening. That is not a defect of the data; it is the definition of the α level applied 7,500 times.

Detect

• Count. Multiply the number of channels by the number of time points, multiply by α, and compare with the number of significant points you found. If they are similar, you have measured your α level. - Look at the baseline. Significant points before the stimulus, where the conditions cannot yet differ, are chance points by construction. - Run the analysis under the null. Sign-flip the subject-lev…

Fix

• Reduce the search space before testing. A single pre-specified measurement in a fixed window at fixed electrodes needs one test and no correction (L3.3, L3.7). - Use a cluster-based permutation test when the window cannot be pre-specified. Because the null is a distribution of maxima over the entire searched space, one p-value covers the whole search and the family-wise error rate is controlled…

Full entry with example →

Pitfall

Cluster p-values read as latency/location claims

Symptom

"The effect began at 312 ms" from a cluster test.

Cause

A cluster-based permutation test performs one test of one hypothesis: that the condition labels are exchangeable across the entire tested set of channels and time points. The null distribution is built from the largest cluster statistic found anywhere in each permuted map, which is exactly what makes the single p-value valid over the whole search — and exactly what prevents it from saying anythin…

Detect

• Read the claim back against the test. Any sentence containing an onset, an offset, a duration, a location or a comparison between clusters is not supported by a cluster p-value. - Move the cluster-forming threshold and watch the boundaries. If the reported onset changes by tens of milliseconds across thresholds you would all have accepted, it is a property of the threshold. - Check whether the…

Fix

• Report the cluster as a description, in those words: “the cluster extended over these channels between these latencies, with its largest statistic at …”, followed by the single inferential claim the test supports — that the conditions differ somewhere in the tested space, with the family-wise error rate controlled at α. - Report the full analysis: the tested space, the point-wise statistic, the…

Full entry with example →

In other tools

In other toolsEEGLAB · FieldTrip — names only

The equivalents of what this lesson does, for a reader who works in another toolbox. Function names only: their own documentation is the place to learn how to call them.

EEGLAB

• statcondEEGLAB
• fdrEEGLAB
• std_statEEGLAB
• limo_eegLIMO plugin (install separately)

FieldTrip

• ft_timelockstatistics(cfg.correctm = bonferroni, fdr, max, cluster)FieldTrip

Names checked 2026-09-18 against EEGLAB 2026.0.0 (plugins at the versions in EEGLAB’s own plugin list) and FieldTrip 20251218.

Reading

1. Pernet et al. (2015). Cluster-based mass univariate simulation study. unverified
2. Smith & Nichols (2009). TFCE. unverified
3. Gelman & Loken (2014). Garden of forking paths. unverified