Real-time processing and neurofeedback

Objectives

• Stream data with LSL (mne-lsl)
• Process in a ring buffer with causal filters and budget latency
• Handle artifacts online
• Design a feedback loop
• State what neurofeedback evidence supports and the controls required (sham, double-blind, CRED-nf)

Why this matters

Here is a perfectly ordinary alpha-feedback pipeline: 160 Hz, an 8–12 Hz band-pass with 129 taps, blocks of 32 samples, ten milliseconds to compute the envelope and draw it. The delay between an event in the participant’s brain and the feedback that reflects it is 610 milliseconds. Six tenths of a second, most of it bought by a filter length nobody chose deliberately, in a loop whose entire premise is that the participant can associate the feedback with something they just did.

Nothing in that pipeline is wrong. The filter is a reasonable filter, the block is a reasonable block, and the number is arithmetic. What is wrong is not knowing the number, because every part of the design that matters — how sharp a filter you can afford, how short a block, whether zero-phase filtering is on the table at all — is a trade against it.

Concepts

Streaming: what LSL actually gives you

The Lab Streaming Layer is the common transport in this field. What it provides that matters for a pipeline:

• Named streams with a declared type, channel count and nominal sampling rate, discoverable on the local network, so the analysis does not have to know which machine the amplifier is on.
• Per-sample timestamps in a common clock, with clock-offset estimation between machines, which is the part people underuse. If the stimulus presentation and the EEG are separate streams, their alignment is a property of those timestamps, not of the order the packets arrived in.
• Buffering with explicit loss semantics: an inlet has a finite buffer and will tell you what it dropped rather than silently interpolating.

mne-lsl is the binding this lesson’s notebook uses, and its main service is an object that behaves like a Raw over a moving window, so the analysis you wrote offline is close to the analysis you run online.

TODO(confirm): specific mne-lsl class and method names are not quoted here; they were not verified against that package’s documentation at a named release in this repository. The notebook prints the package version it ran with, and the concepts above are properties of LSL itself rather than of any binding.

Two things to record in the run log of any streaming experiment, because they cannot be recovered afterwards: the nominal versus effective sampling rate (amplifiers drift; LSL reports both), and every dropped sample, with its timestamp.

The ring buffer, and why it is the natural shape

A real-time pipeline keeps a fixed-length circular buffer of the most recent samples and overwrites the oldest. Its length is set by what the analysis needs — a spectral estimate over two seconds needs two seconds in hand — and it is separate from the block, which is how many new samples arrive before the pipeline runs again.

Two consequences that trip people up:

• The buffer length sets the analysis window, not the latency. A 4-second buffer feeding a Welch estimate every 200 ms does not delay the output by 4 seconds; it means the output describes the last 4 seconds. The latency is the block plus the filter plus the compute.
• Filter state must cross block boundaries. Filtering each block as an independent recording restarts the filter’s memory every block, and the result is a transient at every boundary — pf-filter-across-boundaries, but arriving five times a second forever instead of once at a file join. The fix is to carry the filter state (scipy.signal.sosfilt with zi, or any streaming implementation) so the filter sees one continuous signal that happens to be delivered in pieces. Done properly, block-wise filtering equals whole-signal filtering to numerical precision — the block size then changes the timing and nothing else.

The latency budget, in three terms

End-to-end latency is the time between an event in the signal and the feedback reflecting it. It decomposes:

worst case = D/fs + buffer + P
mean       = D/fs + buffer/2 + P

where D is the filter’s group delay in samples, fs the sampling rate, and P the processing delay you measured. Three separate causes, three separate numbers, and three different things you would do about them.

The buffer term needs its convention named. A block of B samples can be counted two ways, and they differ by exactly one sample:

• B/fs, the block period — the convention this lesson quotes. A sample is not available until its acquisition period completes, so the block is handed over at the end of the last sample’s period and the oldest sample in it has waited a full block. At 32 samples and 160 Hz that is 200.00 ms.
• (B − 1)/fs, first-to-last — the difference between the arrival instants of the first and last samples of the block: 193.75 ms. A real quantity, and not the wait.

The widget prints both, defaulting to the block period. Naming the convention is not pedantry: it is the same discipline L4.2 applies to wavelet width, and an unnamed latency figure is ambiguous by one sample before anyone has disagreed about anything.

A third convention exists and is deliberately not budgeted here: adding a further B/fs for the hold until the next update. That answers “how stale is the number on the screen right now”, which is a fair question about a display and a different question from “how late does an event appear”.

The worked pipeline, which is the widget’s default state and this lesson’s exercise:

row	samples	ms	why
Filter — group delay	64	400.00	(129 − 1)/2, the same at every frequency
Buffer — block assembly	32	200.00	32/160 Hz, block-period convention
Processing — stated	1.6	10.00	your measurement, not derived
Total, worst case	97.6	610.00
Total, mean over block position	—	510.00	half a block less

On the other buffer convention the same pipeline reads 603.75 ms. The loop updates 5.0 times a second.

Only the filter row is a physical fact about the signal processing; the buffer row is arithmetic and the processing row is a number you must go and measure, because no tool can know what your compute, transport and display cost. The widget says so in the row rather than offering a plausible default.

Causal filters, and what the length buys

A linear-phase FIR delays every frequency by exactly (N − 1)/2 samples, so length and latency are the same dial. What the length buys is the transition band: at 160 Hz a 129-tap band-pass has a transition of about 3.3 × 160 / 129 = 4.09 Hz, and 257 taps halves that to 2.05 Hz — for another 400 ms. That is the whole trade, stated in one sentence, and it is why real-time pipelines use filters that would look sloppy offline.

An IIR filter buys attenuation far more cheaply in delay, and charges for it elsewhere. On the same 8–12 Hz band at 160 Hz, Butterworth group delay at 10 Hz runs 17.80 / 32.73 / 48.37 / 64.16 samples for orders 2 / 4 / 6 / 8 — order 4 costs 204.54 ms where a 129-tap FIR costs 400. But:

• the delay is not one number. It varies across the pass band, so a read-out must quote the frequency it was read at and show the range beside it. An envelope built from a band whose edges are delayed differently from its centre is distorted, not just late;
• phase is not linear, so the waveform shape changes, which matters if the feedback is tied to phase rather than to amplitude;
• low orders may not meet the stop band you assumed. Order 2 on this band reaches only −21.9 dB at 5 Hz and −13.7 dB at 15 Hz, which is not a band-pass so much as a suggestion.

Zero-phase filtering is not available, and the reason is worth understanding

Forward–backward filtering has no group delay because, in the interior of a record, it is one convolution with a kernel symmetric about the sample it is correcting. Symmetric means it reaches both ways, so the output at time t is computed partly from signal after t. Offline that is free. Live, that signal does not exist yet.

You can still have the zero-phase result online. You just have to wait for it, and the wait is exact: for an N-tap FIR the symmetric kernel spans ±(N − 1) samples, so the look-ahead is N − 1 samples — twice the causal group delay of the same filter. At 129 taps that is 128 samples, 800 ms, which makes the whole pipeline 1010 ms: exactly what you would pay by doubling the filter length instead. Running the filter twice doubles the stop-band attenuation and doubles what it costs in time. There is no version of this where phase is free.

For an IIR the two-sided kernel never ends, so “how far ahead must I wait” only has an answer once you name a tolerance. On these 8–12 Hz Butterworths, measured as how far before an impulse the forward–backward response still exceeds 1 % of its peak, it is 83 / 145 / 186 / 226 samples for orders 2 / 4 / 6 / 8 — that is 518.75 to 1412.50 ms, and every one of them exceeds the causal group delay of the same order. That figure is a truncation criterion, not a bound: a tighter tolerance demands a longer wait.

Which leaves the shortcut people actually reach for: run filtfilt on each buffer as if the buffer were the whole recording. Its filter row is genuinely zero, and it is not a filter you can use. The widget’s loopback probe comes back with a negative delay — an output that leads its input, which nothing live can do — because within the block the filter is reading samples that arrive after the one it is computing. What you have built is an acausal filter over a completed block, dressed as a live one, paid for with a fresh transient at every block edge. That is pf-filter-across-boundaries five times a second.

Handling artifacts when you cannot look at them

Offline you inspect what was removed and re-run. Online you get one pass, so the design changes (L2.7):

• ICA can run online only with a frozen unmixing matrix, estimated during a calibration block and then applied. It cannot adapt, and if the cap shifts the matrix is describing a head that has moved.
• ASR runs naturally forward-only, which is its main practical advantage here; its sliding window is added to the latency budget — a half-second window is half a second.
• SSP projectors are the cheapest option: a fixed matrix multiplication with no latency at all, and a cost that is countable in rank.
• Gate rather than clean, where you can. The most robust online artifact policy is often to detect a bad stretch and suspend the feedback, telling the participant the signal is unusable, rather than to feed them a cleaned estimate of something that was not measurable. A frozen display is honest; a feedback signal driven by a blink is not.

Whatever you choose, the artifact handling is inside the loop, so it is inside the budget.

Designing the loop

Five decisions, each of which a reader of your protocol will want stated:

1. What is the signal? A band, a ratio, a spatial filter’s output, a classifier’s posterior. Give the exact computation, the window it is computed over and the channels, because “alpha at Pz” is not a definition.
2. What is the mapping to the display? The transfer function from signal to whatever the participant sees or hears, including its scale, its smoothing and its clipping. Smoothing is latency; say how much.
3. What is the baseline, and does it move? A threshold fixed from a calibration block, or an adaptive one that tracks the participant. An adaptive baseline means the participant is always near threshold by construction, which makes the feedback feel responsive and makes “improvement” much harder to define.
4. What is the reinforcement schedule? Continuous feedback, or discrete rewards at threshold crossings, and what happens on a miss.
5. What does the participant actually experience? Latency, granularity and the failure modes. A 610 ms loop with a 5 Hz update rate is a specific perceptual object, and whether anybody can learn from it is an empirical question about that object.

What the evidence supports, and the controls that are required

This is the part of the lesson where it would be easy to overclaim in either direction, so the claims here are kept to what is methodological rather than clinical.

Two different claims are routinely merged and must be separated. One: the participants learned to change the targeted signal. Two: the intervention changed an outcome. Each needs its own evidence. A study that shows a symptom improvement without showing that anybody moved the signal has not demonstrated neurofeedback; it has demonstrated an intervention that involved a screen. A study that shows signal change without an outcome has demonstrated learning, which is interesting and is not a clinical claim.

Why an uncontrolled neurofeedback study cannot support either claim. Between the first session and the last, several things change that have nothing to do with contingent feedback: practice on the task, familiarity with the apparatus, regression to the mean in a group selected for extreme scores, the natural course of the condition, the participant’s expectations, and the experimenter’s. A pre-post design cannot separate any of these from the feedback.

The controls the field has converged on, and which §6 names as this lesson’s objective:

• A sham or control condition. Most commonly feedback that is not contingent on the participant’s own signal — yoked to another participant’s recording, or to the participant’s own signal from a different band or a different channel. The control isolates contingency, which is the only ingredient unique to neurofeedback.
• Double-blinding. Neither the participant nor the person interacting with them knows the condition. Blinding checks belong in the report: ask both parties at the end which condition they believe they were in, and report the answers.
• A pre-specified learning outcome and analysis. Registered before the data exist, for the reasons L6.4 gives in full — the number of defensible ways to define “learning” across sessions, blocks and participants is very large.
• Reporting to a common checklist. The CRED-nf checklist ( (Ros et al., 2020) ) exists so that the items above are not left to each author’s discretion, and so that a reader can tell what was and was not done.

The data behind this lesson

• The widget replays a 10-second single-channel trace from ds-eegbci (S001, O1, 160 Hz, no hardware filters, ODC-By 1.0, snippets allowed). 160 Hz and unfiltered is the point: the delay a filter adds is then the only delay in the trace, and nothing was resampled, because resampling would add a filter and therefore a delay to an asset whose entire subject is delay.
• Every latency figure in this lesson comes from w-latency-budget, is unit-tested in the widget, and is cross-checked against SciPy values shipped in the asset: FIR group delays exactly, Butterworth group delays to six decimals, zero-phase look-aheads to the sample.
• A third trace splices five seconds of eyes-open onto five of eyes-closed at t = 5 s with no cross-fade. It is synthetic and labelled so — no subject produced that ten seconds of EEG — and it exists to show two things at once: how late a causal pipeline notices a change in the alpha envelope, and what a filter does at a discontinuity.

Explore

Latency budget — where the delay in a real-time loop comes from

mode: default Open lab page →

Reset Share state Fullscreen

Loading Latency budget — where the delay in a real-time loop comes from…

What to look for

• Read the three rows before the total. Filter, buffer and processing are separate numbers with separate causes, and the exercise asks for the total of a stated pipeline — so it also asks which row you would attack first.
• Double the tap count. The filter row doubles with it: a linear-phase FIR delays every frequency by exactly (N − 1)/2 samples, so length and latency are the same dial. The buffer and processing rows do not move.
• Now change the block length instead, and watch only the buffer row move. The stated processing delay is the one row the widget cannot derive — it is your measurement to supply, and the widget says so.
• Switch the filter to Butterworth. The group delay is no longer one number: the read-out quotes it at a chosen frequency and prints its range across the pass band, because an IIR delays the edges of the band differently from its middle.
• Now compare the two ways of asking for zero phase. "Zero-phase, live" keeps the delay and renames it: the filter reads samples from after the one it is correcting, so live it has to wait for them — N − 1 samples for an FIR, twice the causal group delay of the same filter — and the row is relabelled "look-ahead" rather than dressed up as free. "Zero-phase per block" runs filtfilt on each buffer as if the buffer were the whole recording: the filter row goes to zero, the probe comes back with a negative delay (an output ahead of its input, which nothing live can do), and every block edge grows its own transient (pf-filter-across-boundaries).

Data: ds-eegbci , subject S001, run R02, 50–60 s · license ODC-By-1.0 · DOI 10.13026/C28G6P · labels: algorithmic · a cropped, re-referenced or filtered derivative of the source recording.

The widget opens on this lesson’s pipeline, so the first thing on screen is the 610 ms budget above. Five things to do:

• Read the three rows before the total, and decide which one you would attack first. They have different causes and different remedies, and the total alone tells you nothing about which.
• Double the tap count and watch one row move. Filter length and latency are the same dial; the buffer and processing rows do not care.
• Change the block length instead, and watch a different row move — and the loop update rate with it.
• Switch to Butterworth and notice that the group delay is no longer one number: the read-out quotes it at a frequency you choose and prints its range across the pass band beside it.
• Compare the two ways of asking for zero phase. “Zero-phase, live” keeps the delay and renames it a look-ahead, at N − 1 samples. “Zero-phase per block” sets the filter row to zero and hands you a probe measurement that is negative, plus a transient at every block edge. Both are honest labels for what the pipeline is actually doing.

Filter sandbox, causal mode — the same delay seen on the waveform

mode: causal Open lab page →

Reset Share state Fullscreen

Loading Filter sandbox, causal mode — the same delay seen on the waveform…

What to look for

• A causal FIR delays every frequency by the same (N − 1)/2 samples; read that delay in milliseconds from the latency table.
• Halve the transition band: the FIR doubles in length and its delay doubles with it.
• Switch to IIR: the group-delay curve is not flat — frequencies near the cutoff are delayed more than those in the middle of the passband.
• Zero-phase filtering is greyed out: it needs future samples, so it cannot run on a live stream.
• Hover the time trace: the filtered series lags the raw one by roughly the delay in the read-out.

Data: ds-eegbci , subject S001, run R01, 36–46 s · license ODC-By-1.0 · DOI 10.13026/C28G6P · labels: algorithmic · a cropped, re-referenced or filtered derivative of the source recording.

The sandbox shows what the budget’s filter row looks like in the signal: the filtered trace shifted against the raw one, and what happens to that shift when you change the design.

• Watch the output lag the input and measure the lag against the group delay the design predicts. They agree, which is the point: group delay is not an abstraction, it is where the wave is.
• Turn zero-phase on and off on the same design and see the shift appear and disappear — then remember that the zero-phase version is the one that cannot run live.
• One warning about quoting the two widgets together. They agree by construction for a linear-phase FIR — both use (N − 1)/2 — but they are controlled differently: the sandbox’s control is the transition band and it derives the tap count from it, while the latency widget’s control is the tap count and it derives the transition band as 3.3·fs/N. Quote one or the other; a mixture will not reconcile.

Practice

A simulated real-time pipeline: the end-to-end latency budget derived, measured with a loopback probe and checked against the widget, then an alpha-envelope feedback loop with sham controls and the false-positive rate of the naive within-session test nb-7-3-realtime

Level 7 ~3 min

notebooks/L7/nb-7-3-realtime.ipynb

Downloads from ds-eegbci.

Open in Colab Download Read it here

The notebook builds a simulated real-time pipeline over replayed ds-eegbci data: a ring buffer, a causal band-pass with its state carried across blocks, an alpha envelope, and a feedback signal — plus a sham mode that feeds back a non-contingent signal, so the control condition is part of the implementation rather than an afterthought. It measures its own end-to-end latency by injecting a known event and timing its appearance, which is the only way to get the processing term, and it compares block-wise filtering against whole-signal filtering to show that the two agree when state is carried and diverge at every boundary when it is not.

Exercises

Exercise ex-7-3-latency-budget

Numeric

A feedback loop runs at 160 Hz with a linear-phase FIR band-pass of 129 taps, blocks of 32 samples, and 10 ms of measured processing delay. Using the BLOCK-PERIOD convention for the buffer term (a block of B samples is handed over at the end of the last sample's acquisition period, so the oldest sample in it has waited B/fs), what is the worst-case end-to-end latency in milliseconds?

Your answer ms

Accepted within ±5 ms.

Check

Hint

Exercise ex-7-3-zero-phase-cost

Numeric

Same loop — 160 Hz, 129 taps, 32-sample blocks, 10 ms processing, block-period buffer — but now you insist on the zero-phase result live, so the filter waits for the future samples its symmetric kernel needs instead of delaying the output. What is the worst-case end-to-end latency in milliseconds?

Your answer ms

Accepted within ±5 ms.

Check

Hint

Exercise ex-7-3-per-block-filtfilt

Multiple choice

To get zero phase online, an engineer applies filtfilt to each 32-sample block independently, carrying no state between blocks. The loopback probe reports a NEGATIVE delay. What has been built?

Options

A correctly implemented zero-phase online filter: the negative delay is a measurement artefact of the probe.An acausal filter over each completed block — within a block the output at one sample is computed from later samples of that block — so the filter row is genuinely zero, the block wait is still paid, and a fresh transient appears at every block edge.A causal filter whose group delay happens to be negative because forward–backward filtering cancels the delay twice.The same pipeline as the causal one, differing only in numerical precision.

Check

Hint

Exercise ex-7-3-protocol-flaw

Free response

A protocol reads: 'Participants completed ten sessions of neurofeedback training. Feedback was the amplitude of 8–12 Hz activity at Pz, displayed as the height of a bar updated continuously, with the threshold adjusted each session so that participants received reward approximately 60 % of the time. Alpha amplitude was compared between the first and last session, and symptom scores were collected before and after the programme.' Name the single most serious design flaw, say what it makes uninterpretable, and give the minimum change that would fix it.

Your answer

Reveal model answer

Hint

Pitfalls

In other tools

In other toolsEEGLAB · FieldTrip — names only

The equivalents of what this lesson does, for a reader who works in another toolbox. Function names only: their own documentation is the place to learn how to call them.

EEGLAB

• vis_streamlsl_app_matlabviewer plugin (install separately)

FieldTrip

• ft_realtime_signalviewerFieldTrip (realtime/example)
• ft_realtime_powerestimateFieldTrip (realtime/example)
• ft_realtime_classificationFieldTrip (realtime/example)

Names checked 2026-09-18 against EEGLAB 2026.0.0 (plugins at the versions in EEGLAB’s own plugin list) and FieldTrip 20251218.

Reading

1. Thibault et al. (2016). Self-regulating brain and neurofeedback. unverified
2. Ros et al. (2020). CRED-nf checklist. unverified