reproducibility Level 6Level 7 pf-unreproducible-archive draft

The archive runs for you and nobody else

Symptom. The deposited code runs on the author's machine and fails, or silently differs, anywhere else.

Symptom

Someone tries to re-run a deposited analysis — a reviewer, a collaborator, a student, or the author a year later — and one of three things happens. It fails to start, because a package is missing or a version has moved. It starts and crashes partway, because a path or a data file is not where the script expects. Or, the expensive case, it runs to completion and produces different numbers, and nobody can say which run was right.

The giveaway is that none of this appeared before deposit. The archive was assembled on a machine where everything already worked, and that is the only machine it has ever been tested on.

Cause

A working analysis is the code plus the environment it ran in, and only the code gets deposited. Four things go missing, in increasing order of how often:

  • Pinned versions. “Analysed in MNE-Python” dates a result without reproducing it. MNE, EEGLAB, FieldTrip, scikit-learn and every statistics package change defaults and behaviour between releases, and a changed default is the kind of difference that produces a plausible number rather than an error.
  • The commit. Code “available on request” or linked to a repository’s main branch is not the code that produced the numbers. Only a commit hash or a tag is.
  • Seeds. Anything with a random component — ICA initialisation, permutation tests, cross-validation folds, surrogate data — gives a different answer on each run unless the seed is recorded with the result.
  • The derivatives. If the figures are drawn from intermediate files, a reader who cannot regenerate those files cannot check the figures, however complete the script is.

The reason this is a pitfall and not merely an omission is that the failure is silent and delayed. Nothing breaks at submission. The analysis works for the person who wrote it, right up until the first time it is run somewhere that installs only what was declared — which is usually long after anyone remembers what the undeclared thing was.

Detect

  • Re-run it from the archive alone, in a clean environment, on a machine that has nothing installed for unrelated reasons. A fresh virtual environment or container is the whole test. Passing on the development machine proves nothing, because that machine is the one place the gaps are filled in.
  • Grep your own methods section for the four items above. If any is absent from the archive, it is absent, whatever you remember doing.
  • Re-run twice and compare. Any difference between two runs of the same archive on the same data is an unrecorded random component.
  • Check that the numbers in the paper come out, not merely that the code exits zero. A pipeline can succeed and be wrong; see the example below.

Fix

  • An environment file with pinned versions, deposited beside the code. A lock file is better than a requirements list, and either is far better than prose.
  • A commit hash or tag in the paper, pointing at the state that produced the numbers.
  • Seeds recorded with the results, not only set in the code.
  • Data in BIDS where it can be shared, with a data-availability statement naming the access route and the licence. Where it cannot be shared, say why, and deposit the derivatives that can be.
  • Treat a clean-environment run as part of submission, in the same way a figure gets checked before it goes in. It is the only step that tests the archive rather than the author’s machine.

Example

This site’s own continuous integration provides the worked case, which is more honest than a hypothetical.

The pipelines job had failed on every run since it was added — 17 failures and 52 errors — and all of them reduced to one missing package. pipelines/pyproject.toml never declared scikit-learn, and MNE implements method='fastica' through it; the test suite runs a configuration that selects fastica, so ICA raised ImportError and every later assertion about excluded components, per-condition counts and epochs failed as a consequence rather than on its own merits.

It passed locally throughout. The development machine has scikit-learn installed for the notebooks, which is exactly how a missing declaration hides: the package is present for an unrelated reason everywhere except the one environment that installs only what is declared. The fix was one line in a dependency list; the cost was that a test suite reported nothing useful for its whole life, and nobody could tell, because it had never once passed and so had no green run to regress from.

The silent variant is worse, and this site has one of those too. ds-srm’s EDF headers declare no physical unit, so MNE reads the data as volts when it is microvolts and every amplitude comes out a million times too large. The dataset is self-consistent about it — its channels.tsv says µV — so nothing in the files contradicts anything. mne-bids-pipeline runs on it, succeeds, and writes a report in which every microvolt is wrong, and no configuration setting catches it. An archive that reproduces that report reproduces the error perfectly, which is why “it runs” is not the test and “the numbers come out” is.