Effect sizes, power and precision

Objectives

• Compute effect sizes appropriate to within-subject EEG designs
• Relate SME to precision
• Run simulation-based power analysis using pilot data

Why this matters

A p-value says whether you can reject something. It does not say how big the effect is, how precisely you measured it, or whether the next study has any chance of finding it again — and those three questions are what the next person needs from your paper. This lesson connects them: precision is the standardized measurement error you met in L3.5, effect size is what precision and true variance together produce, and power is what you get when you simulate the whole chain instead of guessing.

Concepts

Effect sizes for a within-subject design, and why the label matters

For a paired design there is more than one standardized effect size, and they are not interchangeable.

• d_z = mean of the per-subject differences ÷ standard deviation of those differences. This is the one a paired t-test implies: t = d_z × √n. It is the right quantity for this design’s power calculation, and it depends on the correlation between conditions — a design where the two conditions are strongly correlated across subjects gives a large d_z from a small raw difference.
• d_av = mean difference ÷ the average of the two conditions’ standard deviations. This ignores the pairing, which makes it roughly comparable with a between-subjects d, and it is usually the smaller number.
• Partial η² or generalized η² for factorial designs; generalized η² is the one that is comparable across designs with different sets of manipulated factors, which is why it is preferred for meta-analysis.

Reporting “d = 0.8” without saying which d is the most common effect-size error in this literature, and it matters because a power analysis built on someone else’s d_av, applied as if it were d_z, will plan the wrong study.

Report the raw effect first. “Targets were 3.6 µV more positive than standards at Pz over 300–600 ms, 95 % CI [2.1, 5.1]” is the sentence a reader can use: it is in the units of the measurement, it can be compared with the published literature on the same component, and it does not change when the between-subject variance of your particular sample happens to be small. The standardized effect size is a second number, for planning, not the primary result.

Precision and power are different quantities

Precision is how well you measured one subject. The standardized measurement error is the standard error of a single subject’s measurement, estimated analytically for a linear measure (the standard deviation of the single-trial scores over the square root of the trial count) and by bootstrap otherwise.

Power is a property of the design and the analysis over the whole sample.

They meet in the denominator. The between-subject variance you actually observe is not the variance of the true subject effects; it is that variance plus measurement error:

var(observed per-subject scores) ≈ var(true subject scores) + mean(SME²)

so noisier measurement inflates the denominator of d_z and costs power without changing anything about the brains being measured. Two consequences follow, and they are the practical content of this section:

• More trials per subject increases power, even though the number of subjects — the unit of analysis — has not changed. This is why the trial-count question of L3.5 is a power question.
• There is a floor. As SME goes to zero the observed variance converges on the true between-subject variance, and no further trials help. Past that point the only way to buy power is more subjects. A simulation shows you where the knee is; intuition does not.

Simulation-based power, and why the closed forms are not enough

A closed-form power calculation for a paired t-test needs one number, d_z, and returns the required n. It is fast, it is correct for the model it assumes, and it cannot represent an EEG experiment: unequal trial counts, per-subject noise that varies by a factor of ten, a measure that is not linear in the data (peak amplitude), rejection that removes different fractions of each condition, a cluster test rather than a t-test, or a between-group comparison with unequal group sizes.

Simulation represents all of it, because it runs the analysis:

1. Specify the generative model — the true effect size in µV, the between-subject standard deviation of that effect, the per-trial noise, the trial counts per condition.
2. Draw a synthetic study at a given (subjects × trials) cell.
3. Run the exact analysis you intend to publish, including the measurement and the correction.
4. Repeat a few thousand times and count the proportion that rejects. That proportion is the power at that cell.
5. Sweep the grid of subjects × trials and read off the contour you need.

Step 3 is the one people skip, and it is the one that makes the answer correct. Power for the analysis you will actually run is not power for a t-test on a clean mean.

Pilot data: the three ways it misleads

The simulation needs parameters, and the tempting source is your own pilot. Three cautions, in order of how much damage they do:

• The winner’s curse. An effect size estimated from a small sample and selected because it reached significance is upward-biased — necessarily so, because the selection retained the draws where noise happened to help. Planning a study on that number plans an underpowered study. The honest alternatives are the smallest effect size you would care about, a lower confidence bound on the pilot estimate, or a distribution of plausible effects with power reported across it.
• “Observed power” is not evidence. Power computed from the effect size you observed is a one-to-one function of the p-value you already reported: it adds no information, and “the null result had low observed power” is a restatement of “p was large”, not an explanation of it. If you need to argue for absence, use an equivalence test or a Bayesian analysis designed for it.
• The pilot’s noise is not your study’s noise. Different session length, different cap, different lab, different participants. Pilot SME distributions are the part of a pilot that generalises best; pilot effect sizes are the part that generalises worst.

Between-subject designs, where the cohort does the damage

The second half of the notebook leaves the within-subject world, and everything gets harder.

ds-iowapd is 100 people with Parkinson’s disease and 49 controls, three minutes of eyes-open rest each, 64 channels at 500 Hz, CC0 and openly downloadable — a genuinely useful pilot for a between-group resting contrast. It is also an object lesson: the patients are 68 male and 32 female against 26 and 23 in the controls, the groups differ in mean age, and every patient was recorded on dopaminergic medication. A between-group difference in beta power in that cohort has at least four candidate explanations before “disease” is reached, and no sample size fixes it.

ds-dortmund is 608 healthy adults aged 20–70 at baseline, CC0, with three-minute eyes-closed and eyes-open blocks. Subsampling it is the cleanest way to see how a required N behaves when you can actually check the answer: draw 20 people, estimate the effect, plan a study, then draw the study from the remaining people and see whether it works. Do that a few hundred times and the winner’s curse stops being an abstraction.

pf-site-device-confound is the pitfall that stalks both. Two amplifiers, two sites, two caps, or a cohort collected over two decades, will produce group differences that have nothing to do with the grouping variable; when acquisition is confounded with group, more subjects buy a more precise estimate of the wrong thing.

What to report

Effect size with a confidence interval, in raw units and standardized, with the standardized variant named. The SME for the reported measurement, per condition, aggregated across subjects. Trial counts per subject per condition after rejection. The power analysis: the generative model, the parameter source, the analysis that was simulated, the number of simulations, the seed, and the grid. And, if the study was planned on a pilot, the pilot’s size and whether its effect estimate was selected for significance.

Explore

This lesson has no widget; the arithmetic is small enough to do by hand, and doing it by hand is what makes the simulation legible afterwards. With a pilot in front of you:

• compute d_z from the per-subject differences, and then d_av from the two condition standard deviations, and see how far apart they are. Decide now which one you will report and label;
• take the per-subject SME values from L3.5 and compute mean(SME²); compare it with the observed variance of the per-subject scores. The ratio tells you what fraction of your denominator is measurement error rather than real between-subject variation;
• from that, estimate what d_z would become if the SME were halved — and how many trials that would take, remembering that SME falls as one over the square root of the trial count;
• write down the smallest effect size you would care about, in µV, before you look at the pilot’s estimate. Write down why. That number, not the pilot’s, is the one to plan on;
• decide what you will do if the study comes out null, and write that down too.

Practice

Effect sizes, power and precision: a simulated subjects-by-trials power surface for the P3, a between-group resting pilot on Iowa PD, and what a small pilot really tells you, measured by subsampling Dortmund nb-6-3-power-sim

Level 6 ~12 min

notebooks/L6/nb-6-3-power-sim.ipynb

Downloads from ds-erpcore, ds-iowapd, ds-dortmund.

Open in Colab Download Read it here

Two halves. The first simulates power for a within-subject ERP effect across a grid of subjects × trials, running the real measurement and the real test at every cell, and prints the contour at 80 %. The second takes a between-group resting contrast on ds-iowapd as pilot data for a between-subjects simulation, then repeats the exercise on ds-dortmund by subsampling 608 participants down to pilot-sized groups, so the required N can be checked against the cohort it came from. A judgment callout on ds-bonn runs the pseudoreplication comparison directly: the same contrast with the segment as the unit and with the person as the unit.

Exercises

Exercise ex-6-3-power-grid

Numeric

From the notebook's subjects × trials power grid: at the trial count the notebook fixes for this read-off, how many subjects are needed to reach 80 % power for the stated effect?

Your answer subjects

No answer key yet — work it out and compare with the notebook.

Check

Hint

Exercise ex-6-3-independent-units

Numeric

A paper compares 200 single-channel segments from the Bonn healthy sets against 200 segments from the patient sets with a two-sample t-test, reporting 398 degrees of freedom. The 500 segments in that corpus come from 10 individuals — 5 healthy, 5 patients. How many independent units does the comparison actually have, counting both groups together?

Your answer independent units

Exact answer required, in independent units.

Check

Hint

Exercise ex-6-3-observed-power

Multiple choice

Your pre-registered test comes out at p = .31. A reviewer asks you to report the observed (post-hoc) power, computed from the effect size you obtained. What does that number tell the reader?

Options

It quantifies how likely the study was to detect the effect, so a low value excuses the null result.Nothing the p-value did not already say: for a fixed test and sample size, observed power is a one-to-one function of the observed p-value, so it is a restatement rather than an explanation.It estimates the probability that the null hypothesis is true.It is the power the study would have had if the true effect equalled the smallest effect size of interest.

Check

Hint

Exercise ex-6-3-plan-the-study

Free response

You ran a pilot: 12 subjects, 40 trials per condition, a P3 amplitude difference that reached p = .04. Write the power-analysis paragraph for the registered study that follows.

Your answer

Reveal model answer

Pitfalls

Pitfall

Site, amplifier or cap confounded with group or time

Symptom

Two amplifiers at two sites with different sampling rates; two cap systems over 20 years; 50 vs 60 Hz mains by country.

Cause

Every acquisition choice leaves a fingerprint in the data (L0.3, L1.3): the hardware bandwidth and filters shape the spectrum, the amplifier’s noise floor sets the high-frequency baseline, the cap’s electrode set and reference change amplitudes and topographies, the sampling rate decides which harmonics are visible and how resampling behaves, and the mains frequency decides where the line and its…

Detect

• Tabulate hardware per subject (site, amplifier, cap, rate, filters, reference, mains) from the dataset description and the file headers, and cross-tabulate it against the group and against the recording date; any cell imbalance is a confound. - Compute the hardware-sensitive measures (spectral slope over the full band, power above the physiological range, line-noise residue, noise floor) and te…

Fix

• Design first: balance groups across sites and devices, or record every group on every setup; record era-matched controls in archives. - Harmonize what can be harmonized (resample to a common rate after proper anti-alias filtering, restrict to the common hardware passband, common reference, common channel subset) and state what cannot (noise floors, cap geometry). - Model setup explicitly (as a…

Full entry with example →

Reading

1. Clayson et al. (2019). Reporting and power in ERP studies. unverified
2. Boudewyn et al. (2018). How many trials. unverified