Inference and Rigor

Level 6 is about the distance between a result and a claim. Six lessons: the multiple-comparisons landscape and what each correction actually controls; mixed models, for the designs whose structure a per-subject average cannot represent; effect sizes, precision and simulation-based power; circularity and analytic flexibility, with a multiverse over 486 defensible pipelines; decoding read as inference rather than as a score; and reporting, which is the only channel through which any of it reaches a reader.

Two numbers set the level’s terms, and both come from the same 486 pipelines run over twenty subjects of one oddball experiment. On data whose null is true by construction, at least one pipeline reaches p below .05 in roughly three simulated datasets out of four, while a single pipeline fixed in advance reaches it in about one in twenty — which is what α promises, and is the check that the construction is sound. On the real contrast the effect is found by 476 of the 486, every one of them positive, with the reported size running from +0.40 to +6.05 µV. Nothing separates those pictures except when the choices were made, and nobody in any of them did anything a reviewer would call fraud. That is the point: the remedy is procedure — pre-specification, orthogonal selection, cross-validation, preregistration, reporting the whole grid — not vigilance.

This level closes the statistics thread that runs from L3.7 (testing an ERP, and what a cluster p-value licenses) through L4.7 (the same machinery over a time-frequency space) into L6.1 and L6.4. Work the lessons in order — L6.6 assumes all five before it — and finish with the capstone, which audits a published paper and then preregisters one analysis before the data are touched.