Getting a stream: one interface for a replayed recording, a synthetic stream with planted faults and a headset; the block contract; the rate measured against the rate claimed; a planted loss counted and verified; drift in milliseconds; a session recorded in the site's own asset format and read back

nb-7-11-stream Level 7 · Applied Electives ~3 min Used in L7.11 · Getting a stream: timestamps, dropped samples, drift, recording

Downloads from ds-eegbci when you run it.

Download the notebook (.ipynb) Outputs below are the ones stored when it was executed — you do not need to run anything to read it.

nb-7-11-stream · Getting a stream: timestamps, dropped samples, drift, recording (L7.11)

Lesson L7.11 · Level 7 · Status draft — for expert review; uncertain points carry TODO(confirm).

A recording is a file; a stream is a promise. The promise is that samples arrive at a stated rate, in order, with nothing missing, on a clock you can align other things to — and every part of it can fail quietly. This notebook opens a stream three ways through one interface (eegloop.sources.Source), then measures the promise instead of trusting it.

  1. One interface, three sources. A replayed recording, a synthetic stream with planted faults, and — if the driver is installed and a headset is paired — a device. The code above the source never changes.
  2. Blocks in microvolts with per-sample timestamps, the block contract the rest of the course builds on.
  3. Effective against nominal rate, and dropped samples counted — on the synthetic stream, checked against the planted truth, which is the only place a count can be verified rather than merely reported.
  4. Drift: two clocks that disagree by parts per million, and what that is worth in milliseconds.
  5. Recording a session in the site's own asset format — Float32 LE with a JSON sidecar — and reading it back through the same interface that reads the site's shipped assets.

Every number below comes from replayed or synthetic data. The SOURCE = 'device' cell prints which path it took and never produces anything that ships (CONTRACTS, Phase 5: no live output is an answer key).

In [1]:
# Setup: dependencies, the shared helpers, the course library, non-interactive plotting, a quiet downloader.
import importlib.util
import json
import logging
import subprocess
import sys
import tempfile
import time
import warnings
from pathlib import Path

_needed = ("mne", "scipy", "matplotlib", "pandas", "pooch")
_missing = [p for p in _needed if importlib.util.find_spec(p) is None]
if _missing:
    _req = next((d / "requirements.txt" for d in (Path.cwd(), *Path.cwd().parents)
                 if (d / "requirements.txt").exists()), None)
    _cmd = [sys.executable, "-m", "pip", "install", "-q"]
    _cmd += ["-r", str(_req)] if _req else ["mne==1.10.2", "pooch>=1.8"]
    subprocess.check_call(_cmd)

_shared = next((d / "_shared" for d in (Path.cwd(), *Path.cwd().parents)
                if (d / "_shared" / "helpers.py").exists()), None)
if _shared is None:
    # No checkout around this notebook -- a downloaded .ipynb, or a fresh Colab kernel. Fetch the shared
    # helpers the site publishes beside it. With a checkout present the walk-up above wins and nothing
    # is downloaded; the modules land in ./_shared/ and are reused on every later run.
    import json as _json
    import urllib.request as _urllib
    _base = "https://eeg.neurokinetikz.com/notebooks/_shared/"
    _shared = Path.cwd() / "_shared"
    _shared.mkdir(exist_ok=True)
    for _name in _json.loads(_urllib.urlopen(_base + "index.json").read())["modules"]:
        if not (_shared / _name).exists():
            _urllib.urlretrieve(_base + _name, _shared / _name)
sys.path.insert(0, str(_shared))
import helpers_l7 as L7
_loop = L7.import_eegloop()   # loop/eegloop on sys.path from any directory of a checkout; returns loop/

import helpers
import eegloop
from eegloop import Block, StreamInfo
from eegloop.check import render_check, run_check
from eegloop.session import Recorder
from eegloop.sources import ReplaySource, SyntheticSource, blocks, open_source
from eegloop.stream import DropAccount, Reblocker, detect_gaps

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import mne

mne.set_log_level("WARNING")
logging.getLogger("pooch").setLevel(logging.WARNING)
warnings.filterwarnings("ignore", category=RuntimeWarning)
plt.rcParams["figure.dpi"] = 72

# Which stream this notebook runs on. The site and CI use 'replay' and 'synthetic'; 'device' opens a paired
# headset through the driver (section 1) and produces nothing that ships.
SOURCE = "replay"
LOCAL_RECORDING = None      # a path to a recording of your own (.bin with its sidecar, or .npz): a second pass, never shipped
DEVICE_KEY = "muse-2"       # a key of the driver's board table; used only when SOURCE == 'device'

DATASET = "ds-eegbci"
OC4 = ["O1", "O2", "T7", "T8"]
B = 32                                            # block length in samples, throughout
_edf_before = set(Path(mne.get_config("MNE_DATA", str(Path.home() / "mne_data"))).rglob("*.edf"))
print(f"eegloop {eegloop.__version__} · mne {mne.__version__}")
eegloop 0.1.0 · mne 1.10.2

1 · One interface, three sources

Everything above a source is device-free because every source emits the same Block: n_channels × n_samples in microvolts, a running sample_index that counts losses, a start time and per-sample timestamps on the source's own clock, dropped_before when the source knows it lost samples before this block, and a flags dictionary that later steps write into. open_source takes one string — 'replay:<file>', 'synthetic[:scenario]' or 'brainflow:<board-key>' — and returns the same interface.

The replayed source here is ds-eegbci S001 R01 on the occipitotemporal-4 sites, turned into a stream. The device cell guards on the driver being installed and on SOURCE == 'device', and prints which path it took.

In [2]:
t0 = time.time()
raw = helpers.load_spine(DATASET, 1, run=1).pick(OC4)
x_uv = raw.get_data() * 1e6
x_uv = x_uv - x_uv.mean(axis=1, keepdims=True)
fs = float(raw.info["sfreq"])
replay = ReplaySource(x_uv, fs=fs, ch_names=OC4, pace="fast", block_samples=B, kind="replay",
                      nominal={"dataset": DATASET, "subject": "S001", "run": "R01", "hardware_filters": "none (catalog)"})
synthetic = SyntheticSource("gaps", duration_s=120.0, pace="fast")     # a planted 64-sample gap at 60 s and 200 ppm of drift

have_driver = importlib.util.find_spec("brainflow") is not None
if SOURCE == "device" and have_driver:
    device = open_source(f"brainflow:{DEVICE_KEY}", block_samples=B)
    print(f"device path: opened {DEVICE_KEY!r} through the driver — nothing from it is stored below")
elif SOURCE == "device":
    device = None
    print("device path requested but the driver is not installed: pip install 'eegloop[brainflow]' — falling back to replay")
else:
    device = None
    print(f"SOURCE = {SOURCE!r}: the device path is not taken (the driver is {'installed' if have_driver else 'not installed'} here)")

for name, src in (("replay", replay), ("synthetic", synthetic)):
    i = src.info
    print(f"{name:10s} kind={i.kind} clock={i.clock} fs={i.fs:g} Hz channels={i.ch_names} nominal={ {k: v for k, v in i.nominal.items() if k in ('dataset', 'scenario', 'hardware_filters')} }")
print(f"({time.time() - t0:.1f} s)")
SOURCE = 'replay': the device path is not taken (the driver is not installed here)
replay     kind=replay clock=sample fs=160 Hz channels=('O1', 'O2', 'T7', 'T8') nominal={'dataset': 'ds-eegbci', 'hardware_filters': 'none (catalog)'}
synthetic  kind=synthetic clock=device fs=256 Hz channels=('TP9', 'AF7', 'AF8', 'TP10') nominal={'scenario': 'gaps'}
(0.2 s)

2 · Blocks: microvolts, a sample index, timestamps

Read the first few blocks from each source and print the contract's fields. The replayed source stamps its samples on the recording's sample clock; the synthetic source carries per-sample timestamps that run 200 ppm fast against its sample clock — a planted fault, the subject of section 4.

In [3]:
def first_blocks(src, n=3):
    src.start()
    out = []
    try:
        for k, b in enumerate(blocks(src, B)):
            out.append(b)
            if k + 1 >= n:
                break
    finally:
        src.stop()
    return out

for name, src in (("replay", replay), ("synthetic", synthetic)):
    print(f"--- {name}")
    for b in first_blocks(src):
        ts = b.timestamps_s
        print(f"seq {b.seq:3d}  sample_index {b.sample_index:5d}  t_start {b.t_start_s:8.5f} s  shape {b.data.shape}  "
              f"dropped_before {b.dropped_before}  ts[0]={'none' if ts is None else f'{ts[0]:.6f}'}  "
              f"µV range {b.data.min():7.1f} … {b.data.max():6.1f}")
--- replay
seq   0  sample_index     0  t_start  0.00000 s  shape (4, 32)  dropped_before 0  ts[0]=none  µV range   -90.9 …   77.2
seq   1  sample_index    32  t_start  0.20000 s  shape (4, 32)  dropped_before 0  ts[0]=none  µV range  -114.4 …   58.2
seq   2  sample_index    64  t_start  0.40000 s  shape (4, 32)  dropped_before 0  ts[0]=none  µV range   -86.4 …  103.3
--- synthetic
seq   0  sample_index     0  t_start  0.00000 s  shape (4, 32)  dropped_before 0  ts[0]=0.000000  µV range    -8.9 …   31.7
seq   1  sample_index    32  t_start  0.12502 s  shape (4, 32)  dropped_before 0  ts[0]=0.125025  µV range    -3.7 …   40.6
seq   2  sample_index    64  t_start  0.25005 s  shape (4, 32)  dropped_before 0  ts[0]=0.250050  µV range    -8.0 …   36.2

A block's sample_index is the source's own count of samples including the ones it lost: the difference between two blocks' indices is elapsed samples on the source's clock, not samples delivered. That is the fact that lets a filter know how much time passed and a calibration cut an epoch in the right place after a loss.

3 · Effective rate and dropped samples, checked against the planted truth

The synthetic gaps scenario plants one loss — 64 samples at 60 s — and 200 ppm of clock drift, and records both in its truth. Run the library's acceptance report on it, then check the report's two numbers against what was planted. A count that is merely reported is what a driver gives you; a count that is verified is what a planted answer gives you, and only the second can test the accounting.

In [4]:
truth = synthetic.truth
print("planted:", {"gap": truth["gap"], "drift_ppm": truth["drift_ppm"], "fs": truth["fs"]})
report = run_check(SyntheticSource("gaps", duration_s=120.0, pace="fast"), seconds=119.0, mains_hz=60.0)
for r in report["rows"]:
    if r["item"] in ("sampling rate", "dropped samples", "timestamps", "duration"):
        print(f"[{r['status']:4s}] {r['item']:16s} {r['value']}")

n_planted = truth["gap"]["n_missing"]
ms_planted = truth["gap"]["ms_lost"]
n_counted = report["drops"]["total"]
print(f"\nplanted {n_planted} samples ({ms_planted:.1f} ms of timeline); the stream's accounting counted {n_counted} "
      f"in {len(report['drops']['events'])} event(s) at {[round(t, 2) for t, _ in report['drops']['events']]} s")
assert n_counted == n_planted, "the drop accounting disagrees with the planted truth"
print("the accounting agrees with the planted truth to the sample")
planted: {'gap': {'onset_s': 60.0, 'n_missing': 64, 'index': 15360, 'ms_lost': 250.0}, 'drift_ppm': 200.0, 'fs': 256.0}
[ok  ] sampling rate    255.95 Hz measured from the timestamps, 256 Hz claimed (0.02 % off)
[info] duration         118.8 s = 30400 samples / 256 Hz; 950 blocks of 32
[warn] dropped samples  64 in 1 events (32.3 samples/min)
[info] timestamps       inter-sample interval median 1.00, p5 1.00, p95 1.00, max 65.0 sample periods; 0 % of intervals under 0.1 period; 1 interval(s) over 1.5 periods

planted 64 samples (250.0 ms of timeline); the stream's accounting counted 64 in 1 event(s) at [60.26] s
the accounting agrees with the planted truth to the sample
In [5]:
# The same loss seen three ways: the sample index jumps, the timestamps jump, and the accumulated account.
src = SyntheticSource("gaps", duration_s=120.0, pace="fast")
acct = DropAccount()
idx, t_start, dropped, ts_all = [], [], [], []
src.start()
for b in blocks(src, B):
    idx.append(b.sample_index); t_start.append(b.t_start_s); dropped.append(b.dropped_before)
    if b.dropped_before:
        acct.add(b.t_start_s, b.dropped_before)
    if b.timestamps_s is not None:
        ts_all.append(np.asarray(b.timestamps_s))
src.stop()
ts_all = np.concatenate(ts_all)
gaps_from_timestamps = detect_gaps(ts_all, src.info.fs)
print(f"DropAccount: total {acct.total} samples, {len(acct.events)} event(s), {acct.rate_per_min(120.0):.1f} samples/min")
print(f"detect_gaps on the timestamps: {gaps_from_timestamps}  (index of the first sample after the gap, samples missing)")

fig, axes = plt.subplots(1, 2, figsize=(13, 4))
ax = axes[0]
k = np.arange(len(idx))
ax.plot(k, np.asarray(idx) - k * B, lw=1.2)
ax.set(xlabel="Block number", ylabel="sample_index − (block × B)  (samples)", title="The sample index counts what was lost: a step of 64 at the planted gap")
ax = axes[1]
d = np.diff(ts_all) * src.info.fs
ax.plot(np.arange(d.size) / src.info.fs, d, lw=0.8)
ax.set(xlabel="Time (s)", ylabel="Interval between stamps (sample periods)", ylim=(0, 70),
       title="The timestamps jump by 65 periods where 64 samples are missing")
plt.tight_layout(); plt.show()
DropAccount: total 64 samples, 1 event(s), 32.0 samples/min
detect_gaps on the timestamps: [(15360, 64)]  (index of the first sample after the gap, samples missing)
Figure 1 of notebook nb-7-11-stream, an output plot. The text around it states what it shows and the units of every axis.

4 · Drift: two clocks that disagree by parts per million

The synthetic stream's timestamps run 200 ppm fast against its sample clock. Over two minutes that is 24 ms — six samples at 256 Hz — which is the whole width of an evoked response's peak, and it grows linearly for as long as the stream runs. The report's rate row already showed it: the rate measured from the timestamps is the nominal rate divided by (1 + 200 ppm). Measure the drift directly, then state the arithmetic the exercise asks for.

In [6]:
fs_nom = src.info.fs
n = ts_all.size
k_delivered = np.arange(n)                          # delivered-sample count (the gap removed 64, so add them back after it)
gap_index = truth["gap"]["index"]
k_true = k_delivered + np.where(k_delivered >= gap_index, truth["gap"]["n_missing"], 0)
expected_ts = k_true / fs_nom
drift = ts_all - expected_ts                         # stamp minus the sample clock, seconds
slope_ppm = np.polyfit(expected_ts, drift, 1)[0] * 1e6
print(f"measured drift: {slope_ppm:.1f} ppm (planted {truth['drift_ppm']:g}); at the end of {expected_ts[-1]:.0f} s the stamps lead the sample clock by "
      f"{drift[-1] * 1000:.2f} ms = {drift[-1] * fs_nom:.1f} samples")
print(f"arithmetic for the exercise: {truth['drift_ppm']:g} ppm × 120 s = {truth['drift_ppm'] * 1e-6 * 120 * 1000:.1f} ms")

fig, ax = plt.subplots(figsize=(8, 3.6))
ax.plot(expected_ts, drift * 1000, lw=1.2)
ax.set(xlabel="Time on the sample clock (s)", ylabel="Stamp − sample clock (ms)", title=f"{slope_ppm:.0f} ppm of drift: {drift[-1] * 1000:.1f} ms after two minutes")
plt.tight_layout(); plt.show()
measured drift: 200.0 ppm (planted 200); at the end of 120 s the stamps lead the sample clock by 24.00 ms = 6.1 samples
arithmetic for the exercise: 200 ppm × 120 s = 24.0 ms
Figure 2 of notebook nb-7-11-stream, an output plot. The text around it states what it shows and the units of every axis.

Which clock a cue is on decides whether an epoch is cut in the right place. Lesson L7.15 makes this a parameter of the epoch cutter; the point here is only that the two clocks exist and diverge, and that a stream's own timestamps are the thing to measure, not to assume.

5 · Recording a session in the site's own asset format

Recorder writes what a source delivered — untouched, before any step — as a Float32 little-endian .bin with a JSON sidecar, the spec §4.5 shape every asset on this site uses. A learner's recording and a shipped asset are then the same kind of file, and ReplaySource.from_asset reads both. The sidecar records the channels, the rate, the units, any losses the source reported, and license: private; nothing in it names the machine. Record two minutes of the replayed stream into a temporary directory, read it back, and check the round trip to the sample.

In [7]:
with tempfile.TemporaryDirectory() as tmp:
    src = ReplaySource(x_uv, fs=fs, ch_names=OC4, pace="fast", block_samples=B, kind="replay",
                       nominal={"dataset": DATASET, "hardware_filters": "none (catalog)", "reference": "TODO(confirm)"})
    rec = Recorder(tmp, src.info, name="session", dataset=DATASET, subject="S001", run="R01")
    src.start()
    n_blocks = 0
    for b in blocks(src, B):
        rec.write(b); n_blocks += 1
    src.stop()
    bin_path, side_path = rec.close()
    side = json.loads(side_path.read_text())
    print(f"wrote {bin_path.name} ({bin_path.stat().st_size / 1024:.0f} KB) and {side_path.name}")
    for k in ("channels", "sfreq", "units", "n_samples", "duration_s", "dtype", "layout", "license", "label_source", "gaps", "hardware_filters"):
        print(f"  {k:17s} {side[k]}")
    back = ReplaySource.from_asset(bin_path, pace="fast", block_samples=B)
    back.start()
    y = np.concatenate([b.data for b in blocks(back, B)], axis=1)
    back.stop()
    n = y.shape[1]
    err = float(np.abs(y - x_uv[:, :n]).max())
    print(f"\nread back {n} samples through ReplaySource.from_asset; max |difference| {err:.2e} µV (float32 storage)")
    assert err < 1e-3 and n == n_blocks * B
wrote session.bin (152 KB) and session.json
  channels          ['O1', 'O2', 'T7', 'T8']
  sfreq             160.0
  units             uV
  n_samples         9760
  duration_s        61.0
  dtype             float32-le
  layout            channels x samples, row-major
  license           private — the recorded person's own data; not for redistribution unless they say so
  label_source      none
  gaps              []
  hardware_filters  none (catalog)

read back 9760 samples through ReplaySource.from_asset; max |difference| 1.27e-05 µV (float32 storage)

6 · Your own recording, if you have one

When LOCAL_RECORDING names a file, the same acceptance report runs on it and the figure is titled as yours; when it is None the cell is skipped entirely, so no empty figure ships. On a device, the run command is the one below — a markdown cell, never an output:

EEGLOOP_MAC_ADDRESS=<your-address> eegloop check --source brainflow:<board-key> --seconds 30 --mains 60
In [8]:
if LOCAL_RECORDING is not None:
    mine = ReplaySource.from_asset(LOCAL_RECORDING, pace="fast", block_samples=B)
    rep = run_check(mine, seconds=min(60.0, mine.duration_s), mains_hz=60.0)
    print(render_check(rep, title="Your recording — not shipped"))
else:
    print("LOCAL_RECORDING is None: skipped (set it to a .bin with its sidecar to run this on your own file)")
LOCAL_RECORDING is None: skipped (set it to a .bin with its sidecar to run this on your own file)

7 · The numbers the lesson's exercises ask for

In [9]:
print("nb-7-11-stream — L7.11 numbers (draft; TODO(confirm) at author review)")
print()
print(f"(a) Timeline lost to the planted drop in the synthetic 'gaps' scenario: {n_planted} samples at {fs_nom:g} Hz = {ms_planted:.1f} ms; "
      f"the stream's accounting counted {n_counted} (verified against the planted truth).")
print(f"(b) Drift of {truth['drift_ppm']:g} ppm over 120 s = {truth['drift_ppm'] * 1e-6 * 120 * 1000:.1f} ms (measured: {drift[-1] * 1000:.2f} ms at {expected_ts[-1]:.0f} s).")
print("(c) What to log at acquisition that cannot be recovered afterwards: the source's own timestamps and sample counter (drops and drift), "
      "the nominal rate and the measured rate, the reference, the on-device filter state, the mains frequency, and the cue clock — "
      "see the model answer in the lesson.")
nb-7-11-stream — L7.11 numbers (draft; TODO(confirm) at author review)

(a) Timeline lost to the planted drop in the synthetic 'gaps' scenario: 64 samples at 256 Hz = 250.0 ms; the stream's accounting counted 64 (verified against the planted truth).
(b) Drift of 200 ppm over 120 s = 24.0 ms (measured: 24.00 ms at 120 s).
(c) What to log at acquisition that cannot be recovered afterwards: the source's own timestamps and sample counter (drops and drift), the nominal rate and the measured rate, the reference, the on-device filter state, the mains frequency, and the cue clock — see the model answer in the lesson.

Cleanup

Only what this run fetched is deleted. Nothing was written outside a temporary directory.

In [10]:
_edf_after = set(Path(mne.get_config("MNE_DATA", str(Path.home() / "mne_data"))).rglob("*.edf"))
_new = sorted(_edf_after - _edf_before)
for p in _new:
    p.unlink(missing_ok=True)
print(f"deleted {len(_new)} EDF file(s) this run fetched; kept everything that was already cached")
deleted 0 EDF file(s) this run fetched; kept everything that was already cached