Capstone — Rigor audit

Brief

Take a published EEG paper whose data are open. Audit its analysis against the reporting checklist from L6.6. Then choose one key analysis from it, write a preregistration for your own version of that analysis before you look at the data, commit it so the timestamp is real, run it, and compare.

Three things make this harder than it sounds, and they are the capstone.

The audit has to be fair. You are auditing a paper whose authors shared their data, which is why you can audit it at all — and that is a selection effect running in the paper’s disfavour. An audit that reads as an indictment has failed, whatever it found. The standard is: distinguish “not reported” from “not done”; check the supplementary material, the shared code and the data descriptor before you record an omission; name the checklist you are applying and its publication date; and, for every omission, say what a reader cannot check because of it rather than what it implies about the authors. Most of what you will find is conventional. That is the point of reporting standards and it is not an accusation.

The preregistration has to come first, and be checkable. Writing it afterwards and back-dating it is the failure this deliverable exists to make impossible, so the plan goes into version control before any group-level number exists, and the commit hash goes in the report. That is why “timestamped commit” is in the rubric rather than “the plan was written first”: the second is a claim, the first is evidence.

The comparison has to survive both outcomes. If your result matches the paper’s, that is a result. If it does not, that is also a result, and the interesting work is ranking the candidate explanations — subset, preprocessing, measurement definition, statistical model, and ordinary sampling variability — by how much of the gap each could account for. What you may not do is adjust your analysis until the two agree and report the adjusted version as the pre-registered one.

Choosing the paper

The candidate is a class-A directory entry with a published primary result, not a data descriptor. ds-iowapd is the clear one: Anjum et al. (2024), npj Parkinson’s Disease 10, 6 (doi 10.1038/s41531-023-00602-0), with the data at OpenNeuro ds004584 under CC0 and openly downloadable. It has what the capstone needs — a substantive claim, a cohort, and data you can actually re-analyse.

ds-dortmund, ds-srm, ds-pearl-neuro and ds-aszed are data descriptors. They document a resource rather than test a hypothesis, so there is no primary analysis to audit; if you use one of them, the capstone becomes an audit of a resource, which is a different and easier exercise. You may substitute any other paper whose data are genuinely open and whose primary analysis you can re-run — the requirements are a claim, a method section, and access.

TODO(confirm): this site deliberately does not reproduce the paper’s methods, its numbers or its figures. Read the paper and the data descriptor yourself; everything in your audit must be quoted or cited from them, and the dataset entry on this site is a catalogue record, not a summary of the analysis.

Inputs

• Data. ds-iowapd, “Rest eyes open” (OpenNeuro ds004584 v1.0.0, dataset DOI 10.18112/openneuro.ds004584.v1.0.0, CC0, access open, BIDS). 100 people with Parkinson’s disease and 49 controls; one eyes-open resting recording each, roughly three minutes; a 64-channel active cap on a DC amplifier at 500 Hz; Pz as the online reference, and Iz, I1 and I2 excluded, so 60 channels are analysable; a 0.1 Hz online high-pass; 60 Hz mains; EEGLAB .set format. The cohort’s own caveats are part of the material: patients were recorded on dopaminergic medication, the sex split is 68 male / 32 female in the patient group against 26 / 23 in the controls, the groups differ in mean age, and the group sizes are unequal. TODO(confirm): the dataset entry records size_note as TODO(confirm); three minutes of 64 channels at 500 Hz is about 23 MB per participant as float32, so the full cohort is of the order of 3–4 GB, which you should verify against the OpenNeuro snapshot before you start downloading.
• The paper. Read it before you read your own notes on it. The audit needs the methods section, the supplementary material, the data-availability statement and whatever code was shared.
• The checklist. L6.6, and the community standards it cites. Use the interactive checklist there as the working instrument and paste the completed state into the report.
• The lessons. L6.1 for the search space and the correction; L6.2 for the unit of analysis and for covariates; L6.3 for effect size, precision and the cohort confounds this dataset has in abundance; L6.4 for the preregistration and the multiverse; L6.5 if the paper’s key analysis is a classifier, which for a clinical resting-state paper it may well be.
• Your pipeline. The pipelines/ configuration from C2, as far as it applies to resting data. Every parameter you change from that baseline is named in the report with its reason.

Subset rule and runtime

The capstone notebook runs on a documented subset inside the §11 limit of ten minutes on Colab’s free tier, and scales to the full cohort locally:

• The documented subset is 10–20 participants per group, listed by ID in the notebook’s first cells, selected by a rule that cannot depend on the result — the first n IDs in each group, or every participant passing the pre-specified exclusion rule in ID order. State the rule before you state the IDs.
• On the number itself. §11 sets the documented subset at 10–20 subjects, and it is written for a single-cohort capstone. This one contrasts two groups, where that figure has to be read per group or the design collapses. The binding constraint is the one §11 actually enforces — ten minutes on a free Colab tier — and the subject count is a proxy for it. TODO(confirm): whether §11’s figure is meant per group for a two-group capstone is the author’s to settle; C6 and C7 read it the same way in the meantime.
• A FULL_COHORT switch (default False) extends the run to all 149 participants locally. The per-participant function is identical; only the number of rows in the group table changes. Every number you quote says which state it came from.
• Download only the participants you need, cache locally, and delete raw downloads once the per-participant derivatives exist. Check free disk before you start.
• Per-participant derivatives that are expensive to recompute — cleaned data, spectra, per-participant measures — may be precomputed so the group figures redraw without the full download.
• The notebook contains no absolute paths, states every analysis parameter in one place with the seed, and prints in its final cell every number the report quotes.

Deliverables

1. The audit, item by item against the checklist, in a table with three columns: the item, what the paper states (quoted or cited to a section), and what a reader cannot check if it is absent. Mark each item reported / partially reported / not reported / not applicable, and say where you looked — methods, supplement, data descriptor, shared code. Include the items the paper reports well; an audit that lists only failures is not an audit.
2. The audit’s own limitations, in one paragraph: which checklist items post-date the paper, which of your judgements are contestable, and what you could not check because you do not have the authors’ materials.
3. The preregistration for one key analysis, reproduced verbatim, with the commit hash and the timestamp. Ten to twenty lines, in the structure of L6.4: hypothesis and direction; subset and selection rule; exclusions with numeric criteria; the pipeline with parameters; the measurement with its provenance; the model, the unit of analysis, the correction, α and tails; the covariates you will adjust for and why; what would count as disconfirmation; and what you will do if the analysis cannot be run as written.
4. The pipeline run, with per-participant QC: channels flagged and interpolated, components removed with reasons, percentage of data rejected per group, filter settings, and the run log with package versions and the seed. Report the QC by group, because a cleaning step that behaves differently in patients than in controls is a group difference you have manufactured.
5. The analysis, run as pre-registered. The result with an effect size and a confidence interval, the raw effect in its own units, the test in full, and the non-significant parts reported too.
6. The confound analysis this cohort demands. Age, sex and group are not independent here, and the patients were medicated. Show what the result does when age and sex enter the model as covariates, state plainly that adjustment is a modelling assumption rather than a fix (pf-group-demographic-confound), and say what design would actually settle it. If any acquisition variable differs between groups, treat it as pf-site-device-confound and say so.
7. A robustness report. A small multiverse over the two or three decisions most likely to matter for your measure — reference, filter cutoff, epoch length or artifact criterion — reported as a distribution with your pre-registered path marked (L6.4). Note that the reference is already constrained here: the data were recorded against Pz, so “no re-referencing” is itself a choice with consequences.
8. The comparison with the paper, as a table: the published quantity against yours, with the number of participants behind each and the measurement definition of both, so a reader can see whether the two numbers are the same quantity. Where the definitions differ, say so instead of putting the numbers side by side.
9. One paragraph on divergence, with the candidate explanations ranked by how much of the gap each could account for, and what would distinguish between them.
10. Reproducibility. Package versions, seed, configuration hash, runtime, the commit hash of the preregistration, and what a reader would need in order to re-run the whole thing.

Rubric

• The audit is specific and fair — every finding is tied to a named checklist item and to where you looked; “not reported” is distinguished from “not done”; the supplement, the data descriptor and any shared code were checked before an omission was recorded; items the paper reports well are listed alongside the ones it does not; the standard being applied is named and dated; and no finding is phrased as an accusation about the authors.
• The plan was written before the analysis — the preregistration is in version control with a commit hash that precedes every group-level result in the report, and the hash is quoted.
• Divergences are explained — every departure from the pre-registered plan is listed as a departure, with its reason and with the pre-registered version shown alongside; no parameter was changed after the group result was visible without saying so.
• The subset and its selection rule are stated before the IDs, and the FULL_COHORT state of every quoted number is stated.
• Effect size with an interval and the raw effect are reported; a p-value alone is not a result (L6.3).
• The unit of analysis is correct for the claim, and any repeated measurement within participant is modelled rather than counted (L6.2).
• The correction for multiple comparisons matches the search that was actually performed, and the family is named (L6.1).
• The cohort’s confounds — age, sex, medication state, unequal group sizes — are addressed explicitly, and the limits of covariate adjustment are stated rather than assumed away.
• QC is reported per group, and any group difference in data quality or rejection rate is named.
• The robustness report shows the pre-registered path inside the distribution, and the distribution is described as robustness rather than as a test.
• The comparison table states both measurement definitions.
• The notebook runs top to bottom on the documented subset inside the §11 runtime limit, with no absolute paths, a stated seed, and every quoted number printed in the final cell.
• A reader who has the dataset could reproduce every number from the report alone.

Example report structure

1. The paper. What it claims, in one paragraph, with the citation and the data DOI; what its key analysis is; and why you chose it.
2. The audit table, by checklist stage, with the three columns and the four status marks.
3. Audit limitations. The paragraph from deliverable 2.
4. The preregistration, verbatim, with its commit hash and timestamp.
5. Data and subset. Group sizes, selection rule, IDs, FULL_COHORT state, download size, licence and citation.
6. Pipeline and QC, by group.
7. The pre-registered result, in full, with effect size, interval and the test.
8. Confounds, with and without covariates, and what that comparison does and does not settle.
9. Robustness, as a distribution with the pre-registered path marked.
10. Comparison with the paper, with both measurement definitions.
11. Divergence, ranked.
12. Reproducibility, and one paragraph on what you would do differently if you were designing the original study.

Estimated time: TODO(confirm) — the minutes value in the frontmatter is a site-design estimate, not from the spec.