Scalp to Source
Progress

Pitfalls › pf-post-hoc-windows

statistics Level 3Level 6 pf-post-hoc-windows

Windows chosen from the tested data

Symptom. Effect only in the window that looked biggest.

1 claim on this page is unverified. TODO(confirm) marks a specific statement the author has not yet checked against a primary source. Everything else on this page has been reviewed. Treat a marked claim as provisional and go to the cited source rather than quoting the sentence.

Symptom

The effect is significant in the measurement window that was chosen after plotting the condition difference, and not in the window the previous literature used. The window is unusual — narrower than convention, or displaced from the component’s published latency — and its boundaries coincide suspiciously well with where the difference wave crosses zero. The methods section says the window was “determined from the grand average”. Replication attempts using the reported window fail, and the effect moves to a different window in the new sample.

Cause

A measurement window, an electrode set, a measure and a polarity are all selections. Making a selection from the same data you then test means the test no longer has the error rate it claims, because the selection has already searched over alternatives the test knows nothing about.

The mechanism is easiest to see as a maximum. If you consider k candidate windows and take the one with the largest effect, the statistic you report is not a t on a fixed window but the maximum of k correlated t statistics — a quantity whose null distribution is shifted well to the right of the t distribution you compare it against. The inflation grows with the number of windows you would have accepted, and that number is usually far larger than the number you consciously tried: every window you would have been willing to defend counts, whether or not you computed it.

Two features make this particularly easy to fall into with ERP data. First, adjacent time points are strongly correlated, so a noise excursion is always broad and smooth and always looks like a component. Second, the analyst’s freedom is genuinely large — component, window, electrodes, measure, reference, filter, rejection threshold — and each choice is individually defensible, which is exactly the situation that makes the composite indefensible (L6.4, and gelman2014’s garden of forking paths, discussed there).

A related and subtler form: choosing the window from a statistical map of the very contrast being tested, for example measuring inside the boundaries of a significant cluster. The cluster’s extent is a property of the observed data at a chosen threshold (pf-cluster-inference-misread); measuring inside it and testing again is circular.

Detect

  • Read the methods for where the window came from. “Determined from the grand average of the condition difference” is the failure; “determined from the grand average collapsed across conditions” is not.
  • Count the choices you would have accepted, not the ones you made. If more than one window, electrode set or measure would have been defensible, the analysis has degrees of freedom the p-value does not reflect.
  • Compare with the a-priori window. Report the effect in the literature’s window alongside your own; a large divergence is informative either way.
  • Run the selection under the null. Sign-flip the subject-level difference waves, apply the same selection rule — pick the best window — and see how often it produces a significant result. That rate is the analysis’s real false-positive rate.
  • Check across a grid of reasonable choices (a multiverse, L6.4): if the effect exists only at one corner of the grid, it is a property of the corner.

Fix

  • Fix the window in advance, from published work or from your own previous data, and record it before unblinding.
  • Use a collapsed localizer when the component’s latency in your sample is genuinely uncertain: choose the window and electrodes from the average of all conditions together, which under the null carries no information about the condition difference, and then measure the difference there (L3.3).
  • Use an independent localizer — a separate block, a separate task, or a split half — when one is available.
  • Test the whole window at once instead of choosing a sub-window: a cluster-based permutation test over the full epoch and channel set makes the search explicit and corrects for it (L3.7).
  • Pre-register the measurement, or state plainly that the analysis was exploratory and report it as such. An exploratory finding reported honestly is useful; the same finding reported as confirmatory is not.
  • Report the sensitivity: the effect under two or three pre-specified windows, so a reader can see how much the conclusion depends on the choice.

Example

Three panels built from one ERP CORE contrast. Left: a subject-level difference wave with two measurement windows marked — the a-priori window taken from the literature, and the narrower window placed where the observed difference happened to be largest. Middle: the null distribution of the test statistic when the window is fixed in advance, overlaid with the null distribution of the statistic when the same selection rule is applied to sign-flipped data — pick the best window, then test it — showing the second distribution displaced to the right of the first. Right: the resulting false-positive rate at a nominal five per cent level for the fixed window and for the post-hoc window, as a bar pair with the nominal level drawn as a reference line.

TODO(confirm): caption and numbers, once data/scripts/make_figures_p2.py has written /figures/pitfalls/pf-post-hoc-windows.png and its sidecar — including the contrast and subjects used, the two windows, the number of candidate windows the selection rule searches, the number of sign-flip resamples, the seed, and the two false-positive rates. The figure is generated from ds-erpcore subject-level difference waves (CC BY 4.0; derived: epoched, baselined, re-referenced, resampled, averaged per subject). The description above states what the mechanism requires the figure to show; the sidecar’s own alt and caption are the authority once it exists.