SNR, trial counts and design

Why this matters

“How many trials do I need?” is the question that decides whether an experiment can work, and it is usually answered by copying whatever the last paper did. It has a real answer, and the quantity that gives it — the standard error of the measurement you are actually going to report — can be computed from a pilot subject in a few lines. The same quantity makes the difference between conditions in data quality visible, which is where most of the ways an ERP experiment quietly fails are hiding.

Concepts

From signal-to-noise to a design question

L3.2 established that averaging reduces noise as 1/√N. That is a statement about waveforms. What a design needs is a statement about the number you will report: the mean amplitude in your window, the peak latency, the difference between conditions. The noise level of a waveform does not translate into the precision of a measurement without going through the measurement itself, because different measurements respond to noise differently (L3.3).

So the useful quantity is the standard error of the reported measurement, for one subject, in one condition. That is what the standardized measurement error is.

Standardized measurement error

(Luck et al., 2021) defines the standardized measurement error (SME) as the standard error of the value a given subject and condition contributes to the group analysis — in other words, the standard deviation you would see if you could re-run that subject’s session many times and recompute the measurement each time.

For mean amplitude it has a closed form. Compute the measurement on every single trial, take the standard deviation of those single-trial values, and divide by the square root of the number of trials:

aSME = sd( single-trial mean amplitudes ) / sqrt(N)

This works because mean amplitude is linear: the measurement of the average equals the average of the measurements, so the ordinary standard-error formula applies. It is cheap enough to compute for every subject, every condition and every measurement window as a matter of course.

For non-linear measurements — peak amplitude, peak latency, fractional-area latency — there is no closed form, because the measurement of the average is not the average of the measurements. The general estimator is a bootstrap: resample the trials with replacement, recompute the average, recompute the measurement, and take the standard deviation over resamples.

bSME = sd( measurement( average( bootstrap resample of trials ) ) )

Two things follow immediately. The bootstrap SME is defined for any measurement you can compute, which is what makes it general. And comparing the analytic and bootstrap SME for mean amplitude is a useful sanity check that your pipeline does what you think: they should agree.

Aggregating across subjects. Individual SMEs are per subject and per condition; the group-level summary that relates most directly to the noise added to the group analysis is the root mean square of the per-subject SMEs rather than their arithmetic mean. TODO(confirm): the recommended aggregation and its justification should be checked against (Luck et al., 2021) before publication.

Where it belongs. In the methods, per condition, next to the trial counts, for exactly the reason L2.5 gave: unequal SME between conditions is the quantitative form of condition-biased rejection, and it is the number a reader needs in order to judge whether a peak-amplitude difference could be a noise difference (pf-peak-amplitude-noise-bias).

Trial-count curves

Because the SME of a linear measure falls as 1/√N, the curve of SME against trial count is steep at the left and nearly flat at the right. Halving the measurement error costs four times the trials. Set a target — an SME small relative to the between-subject standard deviation of the effect you expect — and read the required N off the curve.

Two adjustments that matter more than the arithmetic:

• The curve is per subject. Noise levels differ enormously between participants, so a design specified from one pilot subject’s curve will be comfortable for some participants and inadequate for others. Compute the curve on several subjects and design for the noisier end.
• Trials and subjects trade off, but not one-for-one. More trials shrink measurement error within a subject; more subjects shrink the standard error of the group mean and also buy generalisability. (Boudewyn et al., 2018) works through this trade-off by simulation and shows that the answer depends jointly on the effect size, the measure, the number of subjects and the number of trials — which is why “the field uses 40 trials” is not an answer.

The design question and the exclusion rule are the same question asked twice. If your design needs N trials per condition for the measurement error you can tolerate, then the pre-specified subject-exclusion rule of L2.5 should drop subjects who fall below it — a rule that is defensible precisely because it was derived before the data existed.

Design confounds

Overlap. When epochs overlap (L3.1), the response to an adjacent event sits inside the epoch and survives averaging if the interval is fixed. Historically this was attacked by estimating the adjacent responses and subtracting them iteratively — the ADJAR family; the modern treatment is linear deconvolution (L3.6). The design-stage fix is cheaper than either: jitter the interval between events so that the overlapping response is smeared across latencies. Overlap becomes a confound rather than a nuisance as soon as the interval distribution differs between conditions — which it does automatically whenever the interval depends on the participant’s response time.

Counterbalancing. If a physical property of the stimuli differs between conditions, the ERP difference is a difference between stimuli, not between conditions. The fix is to rotate stimuli across conditions between participants so that every stimulus appears in every condition across the sample. The same applies to order and to response mapping: a lateralized component compared across two conditions that used different response hands is confounded with motor preparation.

Unequal trial counts. Many of the most useful designs are unbalanced by construction — an oddball paradigm needs the target to be rare, and ds-brain-invaders runs at roughly one target per five non-targets. The rare condition is the interesting one and it is always the noisier average. Report the counts, report the SME per condition, prefer a linear measurement, and consider a count-matched analysis as a check rather than as the primary analysis (L2.5).

Filters and slow components. A high-pass cutoff chosen for cleanliness attenuates slow components and, because offline filters are zero-phase, redistributes the removed energy symmetrically in time — producing opposite-polarity deflections before and after a slow positivity that look like components (pf-hp-cutoff-erp, L1.5 and L2.4). The components most at risk are exactly the ones this level measures most often: the P3 and anything slower. Choose the cutoff from the component, fix it before measurement, and report it.

Eye movements and lateralized effects

Lateralized components are difference waves between hemispheres (L3.3), and so is the electrical consequence of a horizontal eye movement. The eye is a dipole with the cornea positive; a saccade toward a lateral target drives one side of the head positive and the other negative, with a stimulus-locked onset that matches saccade latency and a clean lateralized topography. If saccades are systematically directed toward the target — which is what participants do unless told and monitored otherwise — the artifact has exactly the form of the effect.

The defences, in the order they should be applied: keep targets near fixation and enforce fixation in the design; record horizontal EOG or eye position; detect saccades with a step-function detector rather than an amplitude threshold, because small saccades are steps rather than spikes; reject or correct, applying the same criterion to every condition (L2.5); and then verify on the residual by averaging the horizontal EOG exactly as the effect is averaged — contralateral minus ipsilateral — and showing that no lateralized deflection remains. pf-eye-movements-lateralized has the full checklist.

Subject-level reliability

A last quantity worth computing when the analysis involves individual differences. A measurement’s reliability within a subject — estimated by split-half or by bootstrap — bounds the correlation it can have with anything else: an unreliable measure attenuates every correlation it appears in, so a null correlation between an ERP measure and a behavioural one may be a statement about the measure rather than about the brain. The SME is the per-subject ingredient of that argument, and reporting it makes the attenuation visible rather than mysterious.

The data behind this lesson

• ds-erpcore P3, CC BY 4.0, open access, per-subject downloadable; 30 EEG + 3 EOG channels, 1024 Hz, CMS reference, 60 Hz mains, no software filters, 40 participants per paradigm. TODO(confirm): the author mirrors the ERP CORE entry into the catalogue registry and signs off the dataset page (§10.11 item 8); the shipped asset sidecars also record a licence conflict in the source — the OSF node record says CC BY 4.0, the per-paradigm component’s own LICENSE file says CC BY-SA 4.0 and its dataset_description.json says CC0 — which the author reconciles (§13 item 22).
• The widget’s SME mode works from the same 200 single trials as the averager mode (sub-001, four channels, 40 target against 160 standard), with a per-trial, per-channel pre-stimulus noise estimate, and computes the bootstrap SME in the browser so the curve is a computation rather than a picture. The trial imbalance is the paradigm’s own, which makes the per-condition SME comparison a real one rather than an illustration.
• The notebook computes SME as a function of trial count per subject, for mean amplitude and for peak measures, and prints the trial count at which the SME crosses the threshold the exercise asks about.

Explore

Try this: with mean amplitude selected, compare the bootstrap curve against the analytic reference and confirm they agree; switch to peak amplitude and watch them separate, which is the visual form of “the measurement of the average is not the average of the measurements”; type a target SME and read the required trial count; then widen the measurement window and watch the required count fall, and ask what you gave up to get it. Finish by switching condition and comparing the two curves at the trial counts the experiment actually delivered.

Practice

The notebook computes analytic and bootstrap SME per subject and per condition on ds-erpcore P3 data, plots SME against trial count, and aggregates across subjects. Its final cell prints the trial count this lesson’s exercise asks for.

Exercises