Cluster Permutation Viz
Step through permutations on a real two-condition contrast, watch the null distribution of maximum cluster statistics fill, and read what the resulting p-value does and does not license.
5 claims on this page are unverified. TODO(confirm) marks a specific statement the author has not yet checked against a
primary source. Everything else on this page has been reviewed. Treat a marked claim as
provisional and go to the cited source rather than quoting the sentence.
Modes: erp · tf — the widget below runs in erp. Use Share state to put the exact view in the URL.
What it does
A real within-subject contrast, tested the way a paper would test it, with every intermediate step on screen. The observed t map over channels and time is drawn with the surviving clusters outlined; a player steps through stored permutations, each one a sign-flip relabelling under the null, and the largest cluster statistic of each lands in a histogram; the p-value is the proportion of that null at or above the observed cluster. A toggle draws the uncorrected point-by-point significance map beside the corrected one, and a panel states, in sentences built from the numbers currently on screen, which claims the result supports and which it does not.
The heavy work is precomputed in Python: the observed statistic map, the cluster table, the stored permutation frames that drive the animation and the full null distribution behind the p-value. The browser re-forms clusters when you move the threshold and re-reads the null; it does not re-run the test.
tf mode — time-frequency clusters — is a stub until Phase 3 (§12). The logic is identical; only the tested space changes, from channels × time to channels × time × frequency.
Controls
| Control | What it sets |
|---|---|
| Cluster-forming threshold | The t value a point must exceed to join a cluster (starts at the value the data file was computed with) |
| α | The significance level for the cluster test and for the uncorrected map |
| Play / step / frame | Which stored permutation is on show; 0 is the observed map |
| Frame duration | Animation speed (autoplay is suppressed under prefers-reduced-motion) |
| Uncorrected map | Draw the point-by-point significance map beside the corrected one |
| Cluster outlines | Outline the surviving clusters on the observed map |
| Null source | Use the full stored null, or only the frames you have stepped through |
| Selected cluster | Which cluster the table row, the histogram marker and the licensing panel describe |
What to look for
erp
- Turn the uncorrected map on: points with |t| above the α critical value light up all over the map. At α = .05 roughly 5% of the points light up even where nothing is happening — that is the expected number, not a finding (
pf-uncorrected-timepoint-tests). - Step the player. Each permutation is a relabelling that would make no difference if the null were true; its largest cluster is almost always smaller than the observed one, and the histogram of those maxima is the null the observed cluster is compared against.
- Watch the p-value while the player runs: it is the proportion of the null at or above the observed statistic, so it is coarse early on and settles as the null fills. The value quoted in a paper comes from the full null, not from the first few frames.
- Lower the cluster-forming threshold: clusters merge into fewer, larger ones with bigger t-sums — but the null grows too, because every permutation gets larger clusters. The threshold changes which shapes of effect the test is sensitive to, not how strict it is; the family-wise error rate stays at α either way.
- Read the licensing panel for the selected cluster. The p-value licenses “the conditions differ somewhere in the tested channels and time window”, and it does not license an onset, an offset, a peak channel or a duration (
pf-cluster-inference-misread).
tf
- Time-frequency clusters arrive in Phase 3 (§12); this panel is a stub. Work the
erpmode instead — it shows the whole logic, and only the tested space changes.
Used in
- L3.7 Statistics for ERPs (
erp) - L4.7 Statistics for time-frequency (
tf, Phase 3) - L6.1 The multiple-comparisons landscape (
erp)
Data provenance
ds-erpcore — ERP CORE (Kappenman et al., 2021), open access, per-subject downloadable. Biosemi ActiveTwo, 30 EEG + 3 EOG electrodes in a 10-20 placement scheme, 1024 Hz, CMS online reference, 60 Hz mains, no software filters, 40 participants per paradigm.
The shipped result is the P3 target-minus-standard contrast tested over subject-level difference waves: 19 subjects, 30 channels, 129 time points from −0.199 to 0.801 s at 128 Hz. One subject was excluded by a stated criterion (too few trials surviving the peak-to-peak criterion), and the exclusion is recorded in clusters.json with its reason, as are the per-subject trial counts before and after rejection. The assets are the observed t map, the boolean masks of the candidate clusters, 60 stored permutation frames for the animation, and the full null of 1,024 maximum cluster statistics that the p-values are computed against.
clusters.json records the exact call — mne.stats.permutation_cluster_1samp_test with the cluster-forming threshold, the permutation count, the tail, the seed and the channel adjacency built by mne.channels.find_ch_adjacency on the 30-channel montage — together with the preprocessing chain and every candidate cluster with its own p-value, including the ones that did not survive. That is deliberate: the tail of the distribution is part of the lesson. Generated by data/scripts/make_cluster_fixtures.py and registered in data/manifest.json. The frame’s provenance line is the authority.
Note that the smallest attainable p-value here is 1/1024 ≈ .00098, because MNE counts the identity relabelling as one of the permutations. A p-value at the floor means “smaller than the null’s resolution”, not “essentially zero”.
TODO(confirm): the asset sidecar records a licence conflict in the source — the OSF node record for thsqg says CC BY 4.0, the per-paradigm component’s own LICENSE file says CC BY-SA 4.0, and its dataset_description.json says CC0. The site states CC BY 4.0 on the strength of the node record; the author reconciles the three (§10.11 item 8, §13 item 22) and mirrors the entry into data/registry.yaml. TODO(confirm): the dataset DOI and paper DOI are TODO(confirm) in data/directory.yaml. TODO(confirm): the subject-exclusion criterion and the per-trial rejection flags are label_source: algorithmic until the author reviews them (§4.5). If the real assets are ever absent the widget falls back to a seeded synthetic demonstration and says so — a synthetic fallback is a teaching illustration, not a result.
Open the code
site/src/components/widgets/w-cluster-permutation-viz/ — Widget.svelte, index.ts, meta.ts, stats.ts, licensing.ts, README.md. Repository link: TODO(confirm) (GitHub org/repo, §13 item 3).