preprocessing Level 2 pf-condition-biased-rejection

Rejection that differs by condition

Symptom. One condition loses 30% of trials; effect appears or disappears.

Symptom

The per-condition trial counts after rejection are badly unequal — one condition keeps most of its trials, the other loses a large fraction. An effect appears, grows or reverses as the rejection threshold moves, while both settings still look like reasonable cleaning. A peak amplitude is larger in the condition with fewer surviving trials. In a group analysis, the subjects with the largest effect are the subjects with the largest trial-count imbalance.

Cause

A fixed rejection criterion is not condition-neutral, because the conditions are not equally likely to trip it. Rare, attended or task-relevant stimuli provoke more blinks and more movement; conditions differ in trial length, in arousal, in how often the participant responds. So the same threshold rejects more trials in one condition than the other.

The consequence is not merely a loss of power. The noise in an average falls as the square root of the trial count, so the two condition averages end up estimated with different precision — and several standard ERP measurements are biased by noise in a direction, not just made imprecise. Peak amplitude is the clearest case: picking the maximum of a noisier waveform returns a larger value, so the condition with fewer trials gets a larger peak for purely statistical reasons (pf-peak-amplitude-noise-bias). A “condition difference” is then a difference in trial counts.

There is a second mechanism on top of the first. Because the threshold is usually chosen while the analyst can see its effect on the result, it becomes an analysis degree of freedom: a small move that improves the effect is very hard to distinguish, from the inside, from a small move that improves the cleaning (L6.4).

Detect

  • Build the per-condition rejection table for every subject, every run: trials presented, trials rejected, percentage, and the criterion and channel responsible. epochs.drop_log holds all of it.
  • Plot the percentage rejected per condition against the threshold, and look at the difference between the two curves rather than the total.
  • Compare the effect computed on the full trial set with the effect computed on count-matched subsamples (seeded). A difference that survives matching is not a trial-count artifact; one that does not, is.
  • Report the standardized measurement error per condition (L3.5). Unequal SME across conditions is the quantitative form of this problem.
  • Ask the counterfactual: could you have reached a different threshold by looking at the result? If yes, the threshold was not pre-specified.

Fix

  • Set the criterion condition-blind: estimate it on pooled trials, never per condition, and fix it before looking at the effect. Pre-specify it alongside the subject-exclusion rule.
  • Prefer a data-driven threshold (autoreject, fitted on pooled trials) to a hand-picked one, and report what it chose.
  • Report the per-condition rejection table in the QC report and in the paper — not just the total percentage.
  • Where the imbalance is large, subsample the better-preserved condition to match the other, with a recorded seed, and report the cost in power. This is a stated choice, not a default.
  • Prefer measurements robust to unequal noise: mean amplitude over an a-priori window rather than peak amplitude; and consider correcting or modelling the imbalance rather than hiding it.
  • Where the imbalance cannot be removed, report it as a limitation with the per-condition SME, so a reader can judge what it could have done to the effect.

Example

Three panels from one ERP CORE P3 subject. Left: the percentage of trials rejected in each of the two conditions as a function of the peak-to-peak threshold, two curves that separate as the threshold falls. Middle: the trial counts surviving at a threshold where the separation exceeds twenty percentage points. Right: the difference wave at Pz computed at that threshold and, overlaid, the same difference wave computed with the conditions subsampled to equal trial counts, where the apparent difference in the measurement window is smaller.
ds-erpcore P3 sub-017, epochs with no ocular correction (the case the pitfall is about), 0.1-30 Hz, average reference, baseline (-0.2, 0.0). The criterion is condition-blind — one peak-to-peak threshold over all 30 channels — and it still rejects 35 % of target trials against 14 % of standards at 132.5 µV, a gap of 21 percentage points, because the rare task-relevant stimulus is the one the participant blinks after. The two averages are then estimated with different precision: with all surviving trials the peak of the difference wave at Pz is +1.9 µV; with the standard condition subsampled to the target's 26 trials (seed 20260917) it is +4.3 µV, while the window mean moves only from +0.4 to +1.5 µV. Numbers TODO(confirm) at author review. Data: ds-erpcore · license CC-BY-SA-4.0 · DOI 10.18112/openneuro.ds003069.v1.0.0. Source: data/scripts/make_figures_p2.py.