Reproducible pipelines at scale

Objectives

• Run mne-bids-pipeline on a BIDS dataset
• Containerize and execute in batch (cloud or cluster)
• Track provenance (DataLad or equivalent)
• Publish a reproducibility statement

Why this matters

The pipeline you wrote in L2.8 works on your laptop, on your data, today. This lesson is about the three things that break that sentence: someone else’s laptop, a thousand subjects, and next year. The tools are ordinary — a standard layout, a configuration file, a container, a job queue and a provenance record — and the discipline they enforce is the one Level 6 asked for, made mechanical rather than remembered.

Concepts

BIDS is the interface, and that is the whole point

A pipeline that reads your directory layout is a pipeline only you can run. BIDS fixes the layout, the file naming, the sidecar metadata and the events table, so that a tool can find the recordings without being told where they are ( (Appelhoff et al., 2019) , (Niso et al., 2022) ). The practical consequence is that the pipeline stops containing any knowledge of your dataset: what was a script full of paths becomes a configuration file full of choices, and the choices are the part worth reviewing.

Two things this buys immediately. A dataset that validates can be processed by a tool nobody wrote for it — ds-srm, ds-dortmund, ds-iowapd, ds-erpcore and ds-hbn are all BIDS, and the same pipeline reads all of them. And the diff between two analyses becomes a diff between two configuration files, which is a reviewable object in a way that two versions of a script are not.

mne-bids-pipeline: configuration as the deliverable

The pipeline takes a BIDS dataset and a configuration file and produces derivatives plus an HTML report per subject. What is worth internalising is not the key names but the shape: a configuration has to state the dataset and the subset (root, task, sessions, runs, which subjects and which to exclude), the channel handling (types, reference, bad-channel policy), the filtering, the epoching and baseline, the artifact policy (ICA or SSP, rejection thresholds), the conditions and contrasts, and the execution settings (parallelism, cache, output root).

TODO(confirm): the exact configuration key names, their defaults and which steps are enabled by default vary between releases of mne-bids-pipeline, and this site has not pinned or executed a version — take the names from the version you install, pin it, and record the version in your report. nb-7-8-bids-pipeline is where a working config lives; this section is about what a config must decide, which is stable.

Three habits make the difference between a config that runs and a config that is evidence:

• Every non-default value carries a comment saying why. A config is a methods section that executes; an uncommented threshold is an undocumented choice.
• Read the reports, do not just collect them. The per-subject HTML report is the QC record (L2.8), and a pipeline that runs to completion on a subject whose data are unusable has failed silently.
• Run one subject first, completely, and look at everything. Scale multiplies whatever your config does, including the wrong thing.

Containers, and what “runs from a clean environment” means

A pinned requirements file records the packages you asked for. A lock file records the versions that were resolved. A container image records the whole environment — interpreter, libraries, system dependencies, locale — as one artifact with a digest you can quote.

The claim a container lets you make is narrow and worth stating precisely: this image, run on this input, produces this output. It does not make your analysis correct, it does not make it portable to a different architecture without saying so, and it does not survive the image being rebuilt from a floating base tag — which is why the digest, not the tag, is the thing that goes in the report.

The test for “runs from a clean environment” is behavioural, not rhetorical: clone the repository into an empty directory, on a machine that has none of your caches, build the environment from the file in the repository, run the entry point, and compare the outputs. If any step needs something you know and the repository does not say, the claim is false. Do this before you publish, not after a reviewer asks.

Batch execution: what changes when there are a thousand subjects

Most EEG pipelines are embarrassingly parallel at the subject level and serial at the group level, which gives the standard two-stage shape: a per-subject job array producing per-subject derivatives, then a single group job that reads them. Four requirements appear at scale that never came up on a laptop:

• Isolation. One subject’s failure must not stop the array. Failures are collected and reported as a list, with the reason per subject, and the group stage states how many subjects it actually had.
• Idempotence. Re-running a completed subject must be free or harmless. That means outputs are written to deterministic paths and the pipeline can tell what is already done — otherwise a resumed run silently mixes two versions of the code.
• Logs per unit. One log per subject, retained, with the software versions and the configuration hash inside it. A single interleaved log from 200 parallel jobs is not a record.
• Determinism. Every random step — ICA, cross-validation splits, permutation — gets a seed derived from the subject identifier rather than from wall-clock time or job index, so that a re-run of one subject reproduces that subject and not merely the group average.

The site’s own scale story is smaller and makes the same point: this repository builds its assets with scripts that record what they fetched, delete what they downloaded, and write a manifest entry with a checksum for every file that ships. That is provenance at the scale of a teaching site, and the shape does not change when the numbers do.

Provenance: what has to be recorded for a result to be re-derivable

Provenance is the graph from inputs to outputs, with the transformation labelled at every edge. DataLad is the tool most often used for it in this field — it version-controls the data alongside the code and records the command that produced each output — but the requirement is independent of the tool. A result is re-derivable if you have recorded:

1. The input, by dataset DOI and version or snapshot identifier, plus the exact subset used, and the licence it came under.
2. The code, by repository URL and commit hash, at the commit that produced the numbers.
3. The environment, by container digest or lock file.
4. The configuration, by content hash as well as by inclusion.
5. The seeds, for every stochastic step.
6. The outputs, by checksum, with the run’s date and duration.
7. The failures: which units did not complete, and why.

Item 7 is the one most often missing and the one that most changes a reader’s interpretation, because a cohort of 200 that silently became 173 is a different study.

What changes when the dataset does not fit on a laptop

ds-hbn is the case worth studying, and it is instructive precisely because this site derives nothing from it.

• Scale. 3,155 participants aged 5–21 across releases R1–R11, at 129 channels and 500 Hz. §6 records the corpus as ~1.8 TB. TODO(confirm): this site’s own dataset entry carries size_note: TODO(confirm), so the 1.8 TB figure comes from the specification rather than from a measurement recorded here; check it against the accessions before planning storage.
• “The dataset” is eleven datasets. Each release is a separate OpenNeuro accession with its own DOI and its own version — the entry records the mapping, including that the R9 accession skips one identifier in the sequence. Nothing in your code can say “the HBN dataset”; it has to say which accessions at which versions, and a re-run a year later has to say so again.
• Streaming rather than downloading. The releases are served as DataLad / git-annex repositories, so the metadata can be cloned and individual files fetched on demand. A per-subject job fetches its own subject, computes, writes the derivative, and drops the raw file. Planning that flow is the difference between a feasible analysis and a storage request. A downsampled 100 Hz variant exists for lightweight use, which is the other standard move: develop on the small version, run on the full one, and state which produced each number.
• Metadata is the analysis surface. Events are annotated with HED (Hierarchical Event Descriptors) version 8.3.0, so trial selection is a query over annotations rather than a hand-written map of trigger codes — which is what makes a corpus this size analysable by anyone but its creators.
• The channel names are not 10-20 names. They are E1–E128 plus Cz, so the montage mapping is a pipeline step with a failure mode, not a detail.
• The sample is transdiagnostic despite the name, so “healthy brain network” is not a control cohort (L7.6).

The reproducibility statement

One section, at the end of the paper or the repository README, that a stranger can act on. It names the input datasets with DOIs, versions and licences; the code repository and the commit; the environment as a container digest or lock file; the configuration file and its hash; the seeds; the compute the run needed and how long it took; the units that failed and why; and what a reader must supply themselves, such as a data-use agreement you cannot pass on. The last item is what makes the statement honest rather than promotional: say what a reader cannot reproduce, and why.

Explore

No widget. This is a keyboard exercise, and it is best done on a pipeline you already have.

• Take the configuration from C2 and mark every value in it that is a fact about your dataset (paths, task names, channel names, trigger codes) rather than an analysis choice (cutoffs, thresholds, contrasts). The first group is what BIDS removes; the second is what a config keeps. Most scripts do not separate them at all.
• Write down the values that would change if you ran the same analysis on ds-srm instead. Then write down the values that would change if you ran it on 11 accessions of ds-hbn. The second list is mostly about execution, not analysis, which is the shape this lesson claims.
• Take your last result and try to fill in the seven provenance items above from what you currently have. Whatever you cannot fill in is the gap to close first.
• Clone your own repository into an empty directory and try to run it. Time how long it takes to get to the first number.

Practice

A pipeline you configure instead of write: mne-bids-pipeline on an ERP dataset and a resting one, the BIDS layout and event metadata a config cannot supply, the same subject through two pipelines, and the units error no configuration file could catch nb-7-8-bids-pipeline

Level 7 ~4 min

notebooks/L7/nb-7-8-bids-pipeline.ipynb

Downloads from ds-erpcore, ds-srm.

Open in Colab Download Read it here

A configuration for mne-bids-pipeline on ds-erpcore, run on one subject end to end with its report, and then the same pipeline pointed at ds-srm — the pipeline’s own documented resting example, task resteyesc — to show what changes when the dataset changes and what does not. ds-hbn appears as the reading case for scale, and nothing on this site derives from it.

Exercises

Exercise ex-7-8-hbn-accessions

Numeric

You are planning a batch run over the public HBN-EEG releases R1–R11. Your provenance record must name each input by its own DOI and version. How many separate OpenNeuro accessions is that?

Your answer accessions

Exact answer required, in accessions.

Check

Hint

Exercise ex-7-8-provenance-contents

Multiple select

A colleague sends you a repository and says the analysis is fully reproducible. Which of these, if missing, would leave a result that cannot be re-derived exactly?

Options (select all that apply)

The dataset DOI with its version or snapshot identifier, and the subset of participants actually used.The code repository and the commit hash at which the reported numbers were produced.The environment as a container digest or a lock file, rather than a list of package names.The seed for every stochastic step, including ICA and cross-validation splits.A list of the units that failed to complete, with the reason for each.A diagram of the pipeline stages drawn for the paper.

Check

Hint

Exercise ex-7-8-clean-environment

Multiple choice

Your paper will state that the analysis runs from a clean environment. Which procedure establishes that claim?

Options

Re-run the pipeline on your own machine from a fresh working directory and confirm the numbers match.Clone the repository into an empty directory on a machine with none of your caches, build the environment from the file in the repository, run the entry point, and compare the outputs — recording anything you had to supply that the repository does not state.Publish the list of packages you had installed while developing the analysis, taken from the environment you ran it in.Publish a container image built from the project Dockerfile and quote its tag in the paper.

Check

Hint

Batch-run deliverable checklist ex-7-8-pipeline-deliverable

Checklist (self-graded)

§6's exercise for this lesson is to produce a passing configuration and a provenance graph. Work this list against what you actually produced; check an item only if it is true of the artefacts in your repository, not of your intentions. Self-graded; nothing leaves your browser.

Checklist

The dataset passes the BIDS validator, and the validator output is in the repository.The configuration file runs one subject end to end without manual intervention.Every non-default value in the configuration carries a comment giving the reason for it.The per-subject HTML report exists and has been read, not merely produced.The pipeline version is pinned, and the pinned version is the one that produced the reports.The environment is captured as a container digest or a lock file, and the digest or hash is written down.The repository has been cloned into an empty directory on a machine with none of your caches, and the entry point ran there.Per-subject jobs are isolated: one failure does not stop the run, and failures are collected with their reasons.Re-running a completed subject is harmless, and outputs go to deterministic paths.One log per subject is retained, containing the software versions and the configuration hash.Every stochastic step is seeded from the subject identifier rather than from time or job index.The provenance graph names, for every output: input DOI and version, subset, code commit, environment digest, configuration hash and seed.The graph records which units failed and why, and the group stage states how many units it actually had.Every input dataset’s licence is recorded in the provenance, and it permits what you intend to publish.A reproducibility statement exists and says plainly what a reader cannot reproduce, and why.

0 / 15 checked. Self-graded.

Pitfalls

No pitfall for this lesson: Spec §6 lists no pitfall for this lesson (§5.3 exemption).

In other tools

In other toolsEEGLAB · FieldTrip — names only

The equivalents of what this lesson does, for a reader who works in another toolbox. Function names only: their own documentation is the place to learn how to call them.

EEGLAB

• pop_importbidsEEG-BIDS plugin
• pop_studyEEGLAB
• std_precompEEGLAB
• eeghEEGLAB

FieldTrip

• data2bidsFieldTrip
• ft_analysispipelineFieldTrip
• ft_reproducescriptFieldTrip
• qsubcellfunFieldTrip (qsub)

Names checked 2026-09-18 against EEGLAB 2026.0.0 (plugins at the versions in EEGLAB’s own plugin list) and FieldTrip 20251218.

Reading

1. Niso et al. (2022). Open and reproducible neuroimaging. unverified
2. Appelhoff et al. (2019). MNE-BIDS. unverified