Sensor-space connectivity: coherence, imaginary coherence, PLV, PLI and wPLI from one set of Fourier coefficients on ds-lemon eyes-closed, with a surrogate null, the reference dependence, and Granger causality on a simulation where the answer is known

nb-5-2-connectivity Level 5 · Connectivity and Spatial Analysis ~3 min Used in L5.2 · Sensor-space connectivity

Downloads from ds-lemon when you run it.

Download the notebook (.ipynb) Outputs below are the ones stored when it was executed — you do not need to run anything to read it.

nb-5-2-connectivity · Sensor-space connectivity (L5.2)

Lesson L5.2 · Level 5 · Status draft — for expert review; uncertain points carry TODO(confirm).

L5.1 showed that a single generator makes every sensor pair perfectly coherent. This notebook takes the measures that are supposed to survive that — imaginary coherence, PLI, wPLI — and puts them on real eyes-closed resting EEG, side by side with coherence, on the same epochs and in the same band.

What you will do

  1. Load one ds-lemon subject's eyes-closed blocks, fetching only the first few minutes of the recording.
  2. Compute coherence, imaginary coherence, PLV, PLI and wPLI from one set of Fourier coefficients, so that the only thing that differs between them is which part of the cross-spectrum they keep — and check the two available implementations against each other.
  3. Put coherence and wPLI side by side, against distance, and find the pairs that coherence calls connected and wPLI does not. That contrast is the L5.2 exercise.
  4. Test the matrices against a surrogate null built by shuffling epochs between channels.
  5. Change the reference and watch every number move.
  6. Remove one linear component — what ICA cleaning does, structurally — and watch every number move again.
  7. Run Granger causality on a simulation where the answer is known, and then on a simulation of pure zero-lag mixing with unequal sensor noise, where it confidently reports a direction that does not exist.

Data. ds-lemon — LEMON (MPI-Leipzig Mind-Brain-Body), Babayan et al. (2019), A mind-brain-body dataset of MRI, EEG, cognition, emotion, and peripheral physiology in young and old adults, Scientific Data 6, 180308, DOI 10.1038/sdata.2018.308. Licence and access are read from data/directory.yaml in the first cell. From the catalog: Brain Products BrainAmp MR plus with a 62-channel actiCAP (61 EEG + VEOG), 2500 Hz, FCz online reference, 0.015–1000 Hz with no notch, 50 Hz mains, 16 alternating one-minute eyes-closed / eyes-open blocks.

Download policy. The raw release is about 250 MB per subject compressed. The BrainVision data file is multiplexed, so the first N minutes can be fetched with an HTTP Range request and read as a shorter recording — 3.2 minutes is about 60 MB and holds two complete eyes-closed blocks. Everything is deleted in a finally before the notebook goes on.

In [1]:
# Setup: dependencies, the shared helpers, non-interactive plotting.
import importlib.util
import subprocess
import sys
import warnings
from pathlib import Path

# 1. Dependencies are pinned in notebooks/requirements.txt.  Nothing is installed when the pinned
#    stack is already present (local runs, CI); a fresh Colab or Binder kernel installs it once.
#    On Colab, run from a clone of the repository so that notebooks/_shared/ is available
#    (repository URL: TODO(confirm), spec section 13 item 3).
_needed = ("mne", "scipy", "matplotlib", "pooch")
_missing = [p for p in _needed if importlib.util.find_spec(p) is None]
if _missing:
    _req = next((d / "requirements.txt" for d in (Path.cwd(), *Path.cwd().parents)
                 if (d / "requirements.txt").exists()), None)
    _cmd = [sys.executable, "-m", "pip", "install", "-q"]
    _cmd += ["-r", str(_req)] if _req else ["mne==1.10.2", "pooch>=1.8"]
    subprocess.check_call(_cmd)

# 2. Shared helpers, located relative to the working directory -- notebooks/<level>/ or notebooks/ --
#    never through an absolute path.
_shared = next((d / "_shared" for d in (Path.cwd(), *Path.cwd().parents)
                if (d / "_shared" / "helpers_l5.py").exists()), None)
if _shared is None:
    raise FileNotFoundError("start the kernel in notebooks/L5/ (or notebooks/) so that _shared/helpers_l5.py is found")
sys.path.insert(0, str(_shared))
import helpers
import helpers_l5 as L5

# 3. Plotting: Jupyter's default inline backend renders static PNGs through Agg (no windows, nothing
#    blocks); outside Jupyter the helpers select Agg.  Every MNE figure is requested with show=False
#    and each figure cell ends with plt.show().
import matplotlib.pyplot as plt
import numpy as np
import mne
import pooch

mne.set_log_level("WARNING")
pooch.get_logger().setLevel("WARNING")   # no download chatter: it would print local paths
plt.rcParams["figure.dpi"] = 72
print(f"MNE {mne.__version__}; helpers_l5 imported from notebooks/_shared")
print("mne-connectivity available:", L5.have_mne_connectivity())
print(L5.disk_line("disk at the start"))
MNE 1.10.2; helpers_l5 imported from notebooks/_shared
mne-connectivity available: True
disk at the start: 4.35 GB free on the working volume
In [2]:
print("Licences, from data/directory.yaml (never from memory):")
L5.print_licences("ds-lemon", notes=True)
Licences, from data/directory.yaml (never from memory):
  ds-lemon — LEMON (MPI-Leipzig Mind-Brain-Body): licence CC-BY-4.0, access open (data/directory.yaml)
      CC BY 4.0 per the data descriptor; the NITRC/INDI page references an Open Data (PDDL-style) dedication;
      exact dataset terms TODO(confirm) (§13 item 5)

1 · One subject's eyes-closed blocks

The raw recording marks its blocks with a stimulus code every two seconds: S210 through an eyes-closed block and S200 through an eyes-open one, thirty ticks to a one-minute block, sixteen blocks alternating. TODO(confirm): which code means which condition is not stated in data/directory.yaml. The reading used here is inferred from the recording — the catalog documents eyes-closed first, and the first run of ticks is S210 — and the cell below checks it the only way that can falsify it: posterior alpha power should be larger in the blocks called eyes-closed.

In [3]:
SUBJECT = "sub-010002"
MAX_MINUTES = 3.2
BAND = L5.ALPHA_BAND

print(L5.disk_line("disk before the download"))
print()
for k, v in L5.LEMON_PIPELINE.items():
    print(f"  {k:16s}: {v}")
print()

paths = []
try:
    epochs, meta = L5.lemon_condition_epochs(SUBJECT, "eyes-closed", max_minutes=MAX_MINUTES)
    paths = L5.lemon_files(SUBJECT)
    if epochs is None:
        raise RuntimeError(f"ds-lemon could not be fetched: {meta.get('error')}")
    data_uv = epochs.get_data(copy=True) * 1e6          # (n_epochs, n_channels, n_times), uV
    ch_names = list(epochs.ch_names)
    sfreq = float(epochs.info["sfreq"])
    ep_info = epochs.info.copy()
finally:
    print()
    freed = L5.delete_files(paths)
    print(L5.disk_line("disk after the download was deleted"))

print()
print(f"blocks found in the first {MAX_MINUTES:g} min of the recording:")
for b in meta["blocks"]:
    print(f"  {b['condition']:12s} {b['code']}  {b['start_s']:7.1f}-{b['end_s']:7.1f} s  "
          f"({b['n_ticks']} ticks)")
a = meta["alpha_check"]
print()
print(f"falsifiable check on the S210 reading -- {BAND[0]:g}-{BAND[1]:g} Hz power at {a['channels']}:")
print(f"  eyes-closed {a['eyes_closed_uv2_hz']:.3f} uV^2/Hz, eyes-open {a['eyes_open_uv2_hz']:.3f} uV^2/Hz, "
      f"ratio {a['ratio']:.2f}")
print("  a ratio above 1 is consistent with S210 = eyes closed; it does not prove it, and the catalog does "
      "not say. TODO(confirm)")
print()
print(f"{len(data_uv)} eyes-closed epochs of {meta['epoch_s']:g} s x {len(ch_names)} channels at "
      f"{sfreq:g} Hz ({meta['n_dropped']} dropped above {meta['ptp_criterion_uv']:g} uV peak-to-peak)")
disk before the download: 4.35 GB free on the working volume

  source          : raw release (BrainVision), first N minutes by HTTP Range request
  channels        : 61 EEG (FCz, the online reference, is not a data channel); VEOG dropped
  resample_hz     : 250.0
  filter_hz       : (1.0, 45.0)
  filter_note     : FIR, zero-phase, MNE defaults; 1 Hz high-pass because connectivity is estimated per 2-s epoch
  reference       : average over the 61 EEG channels
  reference_note  : the online reference FCz is absent, so the average is over 61 of the 62 nominal sites
  epoch_s         : 2.0
  epoch_note      : one epoch per marker tick, so epochs tile the block without overlap

ds-lemon sub-010002 (mirror id sub-032301): fetching sub-010002.vhdr, sub-010002.vmrk and 60 MB of sub-010002.eeg (the first 3.2 min of 17.0); 62 channels, 2500 Hz, INT_16
  sub-010002.eeg: 59.5 MB in 4 s
  sub-010002 eyes-closed: 60 epochs of 2 s (61 EEG channels at 250 Hz); 0 dropped above 300 uV peak-to-peak

  deleted 59.5 MB of downloaded files
disk after the download was deleted: 4.35 GB free on the working volume

blocks found in the first 3.2 min of the recording:
  eyes-closed  S210      6.4-   66.4 s  (30 ticks)
  eyes-open    S200     68.7-  128.7 s  (30 ticks)
  eyes-closed  S210    130.5-  190.5 s  (30 ticks)

falsifiable check on the S210 reading -- 8-12 Hz power at ['O1', 'Oz', 'O2', 'PO7', 'PO8']:
  eyes-closed 2.815 uV^2/Hz, eyes-open 0.796 uV^2/Hz, ratio 3.54
  a ratio above 1 is consistent with S210 = eyes closed; it does not prove it, and the catalog does not say. TODO(confirm)

60 eyes-closed epochs of 2 s x 61 channels at 250 Hz (0 dropped above 300 uV peak-to-peak)

2 · Five measures, one set of Fourier coefficients

Every measure below is built from the same cross-spectral products $S_{ij}(e,f) = X_i \overline{X_j}$, averaged over epochs. The only difference between them is which part of $S$ they keep:

measure formula survives zero-lag mixing?
coherence $\lvert\langle S_{ij}\rangle\rvert / \sqrt{\langle S_{ii}\rangle\langle S_{jj}\rangle}$ no — mixing is all it sees
imaginary coherency $\mathrm{Im}\langle S_{ij}\rangle / \sqrt{\langle S_{ii}\rangle\langle S_{jj}\rangle}$ yes — zero-lag has no imaginary part
PLV $\lvert\langle S_{ij}/\lvert S_{ij}\rvert\rangle\rvert$ no — it drops amplitude, not lag
PLI $\lvert\langle \mathrm{sign}\,\mathrm{Im}\,S_{ij}\rangle\rvert$ yes
wPLI $\lvert\langle \mathrm{Im}\,S_{ij}\rangle\rvert / \langle\lvert \mathrm{Im}\,S_{ij}\rvert\rangle$ yes, and it discounts near-zero lags

mne-connectivity is an optional dependency. When it is installed the notebook uses it; when it is not, helpers_l5.connectivity_from_fourier computes the same five measures in about twenty lines of NumPy. The cell below runs both when both are available and prints the largest disagreement, so "we used the fallback" is a statement with a number behind it rather than a hope.

In [4]:
METHODS = ("coh", "imcoh", "plv", "pli", "wpli")
for m in METHODS:
    print(f"  {m:6s}: {L5.CONNECTIVITY_NOTES[m]}")
print()

con, info_lib = L5.connectivity(data_uv, sfreq=sfreq, methods=METHODS, fmin=BAND[0], fmax=BAND[1])
print(f"backend used for every number below: {info_lib['backend']}")

con_np, info_np = L5.connectivity(data_uv, sfreq=sfreq, methods=METHODS, fmin=BAND[0], fmax=BAND[1],
                                  backend="numpy")
if info_lib["backend"] != info_np["backend"]:
    print(f"cross-check against {info_np['backend']}:")
    for m in METHODS:
        print(f"  {m:6s}: largest disagreement over all "
              f"{len(ch_names) * (len(ch_names) - 1) // 2} pairs = "
              f"{np.abs(np.abs(con[m]) - np.abs(con_np[m])).max():.3e}")
    print("  the two paths are the same estimator -- one Hann-tapered DFT per epoch, averaged over epochs -- "
          "so this is")
    print("  machine precision rather than agreement by luck.  A fallback that gave different numbers would "
          "not be a fallback.")
else:
    print("mne-connectivity is not installed; the NumPy path in helpers_l5 produced every number below")
CON = {m: np.abs(con[m]) for m in METHODS}
  coh   : coherence |<S_ij>| / sqrt(<S_ii><S_jj>): all of the shared power, lagged or not
  imcoh : imaginary part of coherency: zero for any zero-lag relation, so instantaneous mixing cannot produce it
  plv   : phase-locking value |<exp(i*phi_ij)>|: phase consistency with amplitude removed; still sees zero lag
  pli   : phase-lag index |<sign(Im S_ij)>|: which side of zero the phase sits on, ignoring how far
  wpli  : weighted phase-lag index |<Im S_ij>| / <|Im S_ij|>: PLI weighted by |Im S|, so near-zero lags count less

backend used for every number below: mne-connectivity 0.9.0 (spectral_connectivity_epochs, mode='fourier')
cross-check against numpy fallback (helpers_l5.connectivity_from_fourier):
  coh   : largest disagreement over all 1830 pairs = 2.220e-16
  imcoh : largest disagreement over all 1830 pairs = 5.551e-17
  plv   : largest disagreement over all 1830 pairs = 1.110e-16
  pli   : largest disagreement over all 1830 pairs = 0.000e+00
  wpli  : largest disagreement over all 1830 pairs = 1.110e-16
  the two paths are the same estimator -- one Hann-tapered DFT per epoch, averaged over epochs -- so this is
  machine precision rather than agreement by luck.  A fallback that gave different numbers would not be a fallback.
In [5]:
fig, axes = plt.subplots(1, 5, figsize=(21, 4.2))
for ax, m in zip(axes, METHODS):
    L5.plot_matrix(CON[m], None, title=f"|{m}|  ({BAND[0]:g}-{BAND[1]:g} Hz)", ax=ax, vmin=0,
                   vmax=1.0 if m in ("coh", "plv") else 0.5, cbar_label="dimensionless (0 to 1)")
fig.suptitle(f"ds-lemon {SUBJECT}, eyes closed, {len(data_uv)} epochs x {len(ch_names)} channels: "
             f"five connectivity measures on the same alpha-band cross-spectra (dimensionless)",
             y=1.03, fontsize=11)
fig.tight_layout()
plt.show()   # render the static figure(s) of this cell inline

print("Summary over all pairs (dimensionless):")
print(f"{'measure':>8s} {'median':>9s} {'mean':>9s} {'min':>9s} {'max':>9s}  strongest pair")
for m in METHODS:
    s = L5.matrix_summary(CON[m], m, labels=ch_names)
    print(f"{m:>8s} {s['median']:9.4f} {s['mean']:9.4f} {s['min']:9.4f} {s['max']:9.4f}  "
          f"{s['extreme_pair'][0]}-{s['extreme_pair'][1]}")
Figure 1 of notebook nb-5-2-connectivity, an output plot. The text around it states what it shows and the units of every axis.
Summary over all pairs (dimensionless):
 measure    median      mean       min       max  strongest pair
     coh    0.4178    0.4360    0.0674    0.9955  P8-P6
   imcoh    0.0320    0.0447    0.0000    0.3118  Oz-PO7
     plv    0.3418    0.3649    0.0655    0.9844  T8-TP8
     pli    0.1074    0.1135    0.0296    0.2963  O1-Oz
    wpli    0.1851    0.1967    0.0602    0.6255  O1-Oz

3 · Coherence against wPLI, which is the lesson

The two matrices above are the same data. Coherence is high nearly everywhere; wPLI is low nearly everywhere. Neither is "the right answer" — but only one of them would still be high if the whole recording came from a single generator, and L5.1 is the proof of which.

The exercise asks which pairs coherence flags and wPLI does not, and what that implies. The cell below defines "flagged" against a surrogate null (section 4 builds it), then lists the pairs by how far apart the two measures put them.

In [6]:
d_names, GC_CM = L5.great_circle_cm(ep_info, L5.HEAD_RADIUS_M)
assert d_names == ch_names
dist = L5.upper_pairs(GC_CM)
coh_v, wpli_v = L5.upper_pairs(CON["coh"]), L5.upper_pairs(CON["wpli"])

fig, axes = plt.subplots(1, 3, figsize=(16, 4.2))
axes[0].plot(coh_v, wpli_v, ".", ms=3, alpha=0.4)
axes[0].set_xlabel("coherence (dimensionless)"); axes[0].set_ylabel("wPLI (dimensionless)")
axes[0].set_title("every sensor pair, alpha band", fontsize=9); axes[0].grid(alpha=0.25)
axes[0].set_xlim(0, 1); axes[0].set_ylim(0, 1)
L5.plot_versus_distance(dist, {"coherence": coh_v, "wPLI": wpli_v},
                        title="against distance across the scalp", ax=axes[1],
                        ylabel="dimensionless (0 to 1)")
axes[1].set_ylim(0, 1)
axes[2].hist(coh_v, bins=40, alpha=0.6, label="coherence")
axes[2].hist(wpli_v, bins=40, alpha=0.6, label="wPLI")
axes[2].set_xlabel("value (dimensionless)"); axes[2].set_ylabel("number of sensor pairs")
axes[2].set_title("distribution over all pairs", fontsize=9); axes[2].legend(fontsize=8)
fig.suptitle(f"ds-lemon {SUBJECT} eyes closed, {BAND[0]:g}-{BAND[1]:g} Hz: coherence and wPLI on identical "
             f"epochs ({len(coh_v)} sensor pairs)", y=1.03, fontsize=11)
fig.tight_layout()
plt.show()   # render the static figure(s) of this cell inline

r_cw = float(np.corrcoef(coh_v, wpli_v)[0, 1])
print(f"correlation between the two measures across the {coh_v.size} pairs: r = {r_cw:+.3f}")
print(f"median coherence {np.median(coh_v):.4f}, median wPLI {np.median(wpli_v):.4f}; "
      f"pairs above 0.5: coherence {int((coh_v > 0.5).sum())}, wPLI {int((wpli_v > 0.5).sum())}")
print(f"coherence falls with distance: median {np.median(coh_v[dist < 5]):.3f} under 5 cm, "
      f"{np.median(coh_v[dist > 15]):.3f} over 15 cm")
print(f"wPLI:                          median {np.median(wpli_v[dist < 5]):.3f} under 5 cm, "
      f"{np.median(wpli_v[dist > 15]):.3f} over 15 cm")
Figure 2 of notebook nb-5-2-connectivity, an output plot. The text around it states what it shows and the units of every axis.
correlation between the two measures across the 1830 pairs: r = +0.226
median coherence 0.4178, median wPLI 0.1851; pairs above 0.5: coherence 699, wPLI 7
coherence falls with distance: median 0.785 under 5 cm, 0.497 over 15 cm
wPLI:                          median 0.184 under 5 cm, 0.210 over 15 cm

4 · A surrogate null, so that "connected" is a decision and not an impression

Every measure here is bounded below by zero and biased upward by finite samples, so a small positive number is not evidence of anything until you know what the estimator does when there is nothing there. The surrogate used here shuffles the epoch order independently for each channel. That leaves every channel's own spectrum exactly as it was and destroys only the within-epoch correspondence between channels, which is precisely what every measure above is estimating. TODO(confirm): surrogate choice is a modelling decision, and epoch shuffling is one of several defensible ones (phase randomisation and time-shifting are others); which the course standardises on is for the author.

In [7]:
N_SURROGATE = 200
rng = np.random.default_rng(L5.SEED)


def surrogate_null(data, sfreq, methods, band, n=N_SURROGATE, rng=rng):
    """Max-statistic null: shuffle epochs per channel, recompute, keep the largest value over pairs."""
    n_ep, n_ch, _ = data.shape
    out = {m: np.empty(n) for m in methods}
    for k in range(n):
        shuffled = np.stack([data[rng.permutation(n_ep), c, :] for c in range(n_ch)], axis=1)
        c_k, _ = L5.connectivity(shuffled, sfreq=sfreq, methods=methods, fmin=band[0], fmax=band[1],
                                 backend="numpy")
        for m in methods:
            out[m][k] = np.abs(L5.upper_pairs(np.abs(c_k[m]))).max()
    return out


NULL_METHODS = METHODS
null = surrogate_null(data_uv, sfreq, NULL_METHODS, BAND)
print(f"{N_SURROGATE} epoch-shuffled surrogates; the null is the distribution of the LARGEST value over all "
      f"{coh_v.size} pairs,")
print("so a pair that beats the 95th percentile is significant with the multiple comparisons already paid for.")
print()
thr = {}
for m in NULL_METHODS:
    thr[m] = float(np.percentile(null[m], 95))
    obs = L5.upper_pairs(CON[m])
    print(f"  {m:6s}: null max-statistic median {np.median(null[m]):.4f}, 95th percentile {thr[m]:.4f}; "
          f"{int((obs > thr[m]).sum())} of {obs.size} pairs survive "
          f"({100 * (obs > thr[m]).mean():.1f} %)")
print()
print("Read the coherence row carefully.  This null destroys ALL dependence between channels, so what it tests "
      "is")
print("'is there any relationship at all', and volume conduction guarantees the answer is yes.  It is NOT a "
      "null for")
print("'connected beyond mixing', and no sensor-level surrogate can be: the mixing is in the data, not in the "
      "estimator.")
200 epoch-shuffled surrogates; the null is the distribution of the LARGEST value over all 1830 pairs,
so a pair that beats the 95th percentile is significant with the multiple comparisons already paid for.

  coh   : null max-statistic median 0.2042, 95th percentile 0.2290; 1453 of 1830 pairs survive (79.4 %)
  imcoh : null max-statistic median 0.1471, 95th percentile 0.1709; 27 of 1830 pairs survive (1.5 %)
  plv   : null max-statistic median 0.1966, 95th percentile 0.2163; 1330 of 1830 pairs survive (72.7 %)
  pli   : null max-statistic median 0.2148, 95th percentile 0.2370; 6 of 1830 pairs survive (0.3 %)
  wpli  : null max-statistic median 0.3334, 95th percentile 0.3767; 31 of 1830 pairs survive (1.7 %)

Read the coherence row carefully.  This null destroys ALL dependence between channels, so what it tests is
'is there any relationship at all', and volume conduction guarantees the answer is yes.  It is NOT a null for
'connected beyond mixing', and no sensor-level surrogate can be: the mixing is in the data, not in the estimator.
In [8]:
# The exercise: pairs that coherence calls connected and wPLI does not.
iu = np.triu_indices(len(ch_names), 1)
flag_coh = coh_v > thr["coh"]
flag_wpli = wpli_v > thr["wpli"]
only_coh = flag_coh & ~flag_wpli
print(f"ANSWER KEY -- ex-5-2 (which measure flags a connection that disappears under wPLI):")
print(f"  coherence flags {int(flag_coh.sum())} of {coh_v.size} pairs; wPLI flags {int(flag_wpli.sum())}.")
print(f"  {int(only_coh.sum())} pairs are flagged by coherence and NOT by wPLI "
      f"({100 * only_coh.mean():.1f} % of all pairs); "
      f"{int((flag_wpli & ~flag_coh).sum())} the other way round.")
print()
order = np.argsort(-(coh_v - wpli_v))
print(f"  the ten pairs where the two measures disagree most:")
print(f"  {'pair':>14s} {'distance (cm)':>14s} {'coherence':>10s} {'wPLI':>8s} {'|imcoh|':>9s} {'PLV':>7s}")
for k in order[:10]:
    i, j = iu[0][k], iu[1][k]
    print(f"  {ch_names[i] + '-' + ch_names[j]:>14s} {GC_CM[i, j]:14.1f} {CON['coh'][i, j]:10.4f} "
          f"{CON['wpli'][i, j]:8.4f} {CON['imcoh'][i, j]:9.4f} {CON['plv'][i, j]:7.4f}")
print()
print("  What it implies: a pair with high coherence and wPLI at the null is a pair whose relationship has no")
print("  measurable lag.  That is what one generator seen by two electrodes looks like (L5.1), and it is also")
print("  what a genuine zero-lag interaction would look like -- the measure cannot separate them, so the")
print("  honest reading is 'consistent with mixing', not 'no connection'.  wPLI buys robustness to mixing by")
print("  giving up the ability to see zero-lag coupling at all.")
ANSWER KEY -- ex-5-2 (which measure flags a connection that disappears under wPLI):
  coherence flags 1453 of 1830 pairs; wPLI flags 31.
  1422 pairs are flagged by coherence and NOT by wPLI (77.7 % of all pairs); 0 the other way round.

  the ten pairs where the two measures disagree most:
            pair  distance (cm)  coherence     wPLI   |imcoh|     PLV
           P8-P6            2.5     0.9955   0.0853    0.0018  0.9843
           P3-P5            2.2     0.9921   0.1596    0.0085  0.9823
          P3-CP3            2.5     0.9131   0.0882    0.0030  0.8633
          T8-TP8            3.0     0.9936   0.1704    0.0010  0.9844
          CP3-P5            3.4     0.9058   0.0986    0.0040  0.8493
         CP6-CP4            2.7     0.8925   0.0925    0.0121  0.7878
          CP1-P1            2.5     0.9084   0.1239    0.0263  0.8341
           P4-P2            2.1     0.8782   0.0980    0.0105  0.8056
          FC1-F1            2.4     0.9216   0.1431    0.0196  0.8411
           C4-C6            2.7     0.8903   0.1171    0.0052  0.7692

  What it implies: a pair with high coherence and wPLI at the null is a pair whose relationship has no
  measurable lag.  That is what one generator seen by two electrodes looks like (L5.1), and it is also
  what a genuine zero-lag interaction would look like -- the measure cannot separate them, so the
  honest reading is 'consistent with mixing', not 'no connection'.  wPLI buys robustness to mixing by
  giving up the ability to see zero-lag coupling at all.

5 · The reference moves every number

Connectivity is computed on channel differences, and a reference change subtracts the same signal from every channel. That is not a small perturbation: it adds one common component to every channel, which is exactly the thing every measure here is trying to detect.

ds-lemon's montage has no mastoid electrode, so TP7/TP8 stand in for M1/M2 and the substitution is stated wherever the number is printed — the same convention nb-3-4-topomaps uses for P9/P10 on ERP CORE. TODO(confirm) whether TP9/TP10 exist in other LEMON releases.

In [9]:
# A single-electrode reference makes that electrode identically zero, and a channel with no power has no
# coherence with anything: the estimate is 0/0.  Every reference is therefore compared on the SAME reduced
# channel set -- the 59 channels that are not Cz, TP7 or TP8 -- so the numbers are comparable.
DROP = ["Cz", "TP7", "TP8"]
keep_idx = [i for i, c in enumerate(ch_names) if c not in DROP]
keep_names = [ch_names[i] for i in keep_idx]


def rereference(x, mode):
    if mode == "average":
        y = x - x.mean(axis=1, keepdims=True)
    elif mode == "Cz":
        y = x - x[:, [ch_names.index("Cz")], :]
    elif mode == "linked TP7/TP8":
        y = x - 0.5 * (x[:, [ch_names.index("TP7")], :] + x[:, [ch_names.index("TP8")], :])
    else:
        raise ValueError(mode)
    return y[:, keep_idx, :]


REFS = ["average", "Cz", "linked TP7/TP8"]
ref_con = {}
print(f"the same {len(keep_names)} channels under every reference "
      f"({', '.join(DROP)} dropped, because a reference electrode is identically zero)")
print()
print(f"{'reference':>18s} {'median coh':>12s} {'median wPLI':>13s} {'r with average ref (coh)':>26s} "
      f"{'(wPLI)':>10s}")
for mode in REFS:
    c, _ = L5.connectivity(rereference(data_uv, mode), sfreq=sfreq, methods=("coh", "wpli"),
                           fmin=BAND[0], fmax=BAND[1])
    ref_con[mode] = {k: np.abs(v) for k, v in c.items()}
    rc = float(np.corrcoef(L5.upper_pairs(ref_con[mode]["coh"]), L5.upper_pairs(ref_con["average"]["coh"]))[0, 1])
    rw = float(np.corrcoef(L5.upper_pairs(ref_con[mode]["wpli"]), L5.upper_pairs(ref_con["average"]["wpli"]))[0, 1])
    print(f"{mode:>18s} {np.median(L5.upper_pairs(ref_con[mode]['coh'])):12.4f} "
          f"{np.median(L5.upper_pairs(ref_con[mode]['wpli'])):13.4f} {rc:26.3f} {rw:10.3f}")
print()
print("A reference change cannot change a topography's shape (L3.4) but it can and does change every "
      "connectivity value,")
print("because it changes what each channel IS.  The ranking of pairs survives better than the values do -- "
      "which is why a")
print("paper that reports connectivity without naming its reference has not reported its result.")
the same 58 channels under every reference (Cz, TP7, TP8 dropped, because a reference electrode is identically zero)

         reference   median coh   median wPLI   r with average ref (coh)     (wPLI)
           average       0.4327        0.1883                      1.000      1.000
                Cz       0.3329        0.1743                      0.423      0.452
    linked TP7/TP8       0.3490        0.1582                      0.442      0.686

A reference change cannot change a topography's shape (L3.4) but it can and does change every connectivity value,
because it changes what each channel IS.  The ranking of pairs survives better than the values do -- which is why a
paper that reports connectivity without naming its reference has not reported its result.

6 · Removing one component moves every number too

ICA cleaning removes a component by projecting the data onto the subspace orthogonal to it. Whatever the component means, the operation is a rank-reducing linear projection of the whole channel set — so it changes the cross-spectrum of pairs that had nothing to do with the component. The demonstration below uses the leading principal component rather than an ICA component, because the point is structural and PCA makes it visible without a fitting step: remove one direction, and every entry of the matrix moves.

TODO(confirm): the size of the effect for a real artefact component, on this dataset, is not measured here.

In [10]:
flat = data_uv.transpose(1, 0, 2).reshape(len(ch_names), -1)
U, S, Vt = np.linalg.svd(flat - flat.mean(axis=1, keepdims=True), full_matrices=False)
var = S ** 2 / (S ** 2).sum()
print(f"leading principal component carries {100 * var[0]:.1f} % of the variance across "
      f"{len(ch_names)} channels; the first five carry {100 * var[:5].sum():.1f} %")

proj = np.eye(len(ch_names)) - np.outer(U[:, 0], U[:, 0])
cleaned = np.einsum("ij,ejt->eit", proj, data_uv)
c_clean, _ = L5.connectivity(cleaned, sfreq=sfreq, methods=("coh", "wpli"), fmin=BAND[0], fmax=BAND[1])
c_clean = {k: np.abs(v) for k, v in c_clean.items()}
print()
print(f"{'measure':>8s} {'median before':>15s} {'median after':>14s} {'max |change|':>14s} "
      f"{'r before vs after':>19s}")
for m in ("coh", "wpli"):
    b, a_ = L5.upper_pairs(CON[m]), L5.upper_pairs(c_clean[m])
    print(f"{m:>8s} {np.median(b):15.4f} {np.median(a_):14.4f} {np.abs(a_ - b).max():14.4f} "
          f"{float(np.corrcoef(b, a_)[0, 1]):19.3f}")
print()
print("Every pair moves, including pairs whose channels barely load on the removed component: the projection")
print("is applied to the whole channel set at once.  Reporting how many components were removed is therefore "
      "part of")
print("reporting a connectivity result, not a preprocessing footnote.")
leading principal component carries 26.8 % of the variance across 61 channels; the first five carry 76.1 %

 measure   median before   median after   max |change|   r before vs after
     coh          0.4178         0.3682         0.5833               0.397
    wpli          0.1851         0.1777         0.3346               0.427

Every pair moves, including pairs whose channels barely load on the removed component: the projection
is applied to the whole channel set at once.  Reporting how many components were removed is therefore part of
reporting a connectivity result, not a preprocessing footnote.

7 · Directed measures, and the assumption that breaks them

Granger causality, DTF and PDC all answer a prediction question: does knowing channel $i$'s past reduce the error in predicting channel $j$'s present? Three assumptions are doing the work, and only one of them is about the brain:

  1. the relationship is well described by a linear autoregressive model of the chosen order;
  2. the two channels are observed with comparable noise — unequal SNR alone produces asymmetry;
  3. there is no unobserved common driver — and in sensor-space EEG every generator is a common driver of every channel.

The cells below run the same estimator on three simulations: one where the answer is known and it gets it right, one where the coupling is buried in shared oscillatory dynamics, and one with a single generator, no lag at all, and unequal sensor noise.

In [11]:
def granger(X, a, b, sfreq=250.0, fmin=5.0, fmax=60.0):
    """Mean spectral Granger causality from channel a to channel b, or None when the library is absent."""
    if not L5.have_mne_connectivity():
        return None
    from mne_connectivity import spectral_connectivity_epochs
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        c = spectral_connectivity_epochs(X, method=["gc"], indices=([[a]], [[b]]), mode="multitaper",
                                         sfreq=sfreq, fmin=fmin, fmax=fmax, verbose=False)
    return float(np.asarray(c.get_data()).squeeze().mean())


SIM_FS, SIM_T, SIM_EP = 250.0, 500, 60
r = np.random.default_rng(L5.SEED)
sims = {}

X = np.zeros((SIM_EP, 2, SIM_T))                      # 1: x drives y at one sample's lag
for e in range(SIM_EP):
    x = r.standard_normal(SIM_T)
    y = np.zeros(SIM_T); y[1:] = 0.8 * x[:-1] + 0.3 * r.standard_normal(SIM_T - 1)
    X[e] = [x, y]
sims["A  x drives y, lag 1 sample, no shared dynamics"] = X

Z = np.zeros((SIM_EP, 2, SIM_T))                      # 2: both are the same oscillator, x drives y
for e in range(SIM_EP):
    x = np.zeros(SIM_T); y = np.zeros(SIM_T)
    ex, ey = r.standard_normal(SIM_T), r.standard_normal(SIM_T)
    for tt in range(2, SIM_T):
        x[tt] = 0.55 * x[tt - 1] - 0.8 * x[tt - 2] + ex[tt]
        y[tt] = 0.55 * y[tt - 1] - 0.8 * y[tt - 2] + 0.5 * x[tt - 1] + ey[tt]
    Z[e] = [x, y]
sims["B  x drives y, both strongly oscillatory"] = Z

for label, (g1, g2, n1, n2) in {
    "C  ONE source, zero lag, equal sensor noise": (1.0, 1.0, 0.2, 0.2),
    "D  ONE source, zero lag, UNEQUAL sensor noise": (1.0, 0.3, 0.2, 0.2),
}.items():
    W = np.zeros((SIM_EP, 2, SIM_T))
    for e in range(SIM_EP):
        s = mne.filter.filter_data(r.standard_normal(SIM_T), SIM_FS, BAND[0], BAND[1], verbose=False)
        s = s / s.std()
        W[e] = [g1 * s + n1 * r.standard_normal(SIM_T), g2 * s + n2 * r.standard_normal(SIM_T)]
    sims[label] = W

print(f"Spectral Granger causality, 5-60 Hz, {SIM_EP} epochs of {SIM_T / SIM_FS:g} s "
      f"(multitaper; mne-connectivity)")
print(f"{'simulation':>48s} {'GC 1->2':>9s} {'GC 2->1':>9s} {'coherence':>11s} {'wPLI':>8s}")
gc_rows = []
for label, Xs in sims.items():
    g12, g21 = granger(Xs, 0, 1, SIM_FS), granger(Xs, 1, 0, SIM_FS)
    c, _ = L5.connectivity(Xs, sfreq=SIM_FS, methods=("coh", "wpli"), fmin=BAND[0], fmax=BAND[1])
    row = (label, g12, g21, float(np.abs(c["coh"])[0, 1]), float(np.abs(c["wpli"])[0, 1]))
    gc_rows.append(row)
    if g12 is None:
        print(f"{label:>48s} {'n/a':>9s} {'n/a':>9s} {row[3]:11.4f} {row[4]:8.4f}")
    else:
        print(f"{label:>48s} {g12:9.4f} {g21:9.4f} {row[3]:11.4f} {row[4]:8.4f}")
print()
if gc_rows[0][1] is None:
    print("mne-connectivity is not installed, so the directed measures were not computed in this run; the "
          "symmetric")
    print("measures above came from the NumPy fallback and are unaffected.")
else:
    print("Read row D.  There is ONE generator, there is no lag, and Granger causality reports a strong and")
    print("asymmetric directed influence -- from the high-SNR sensor to the low-SNR one -- because unequal")
    print("measurement noise makes one channel's past a better predictor than the other's.  Row C shows the same")
    print("geometry with equal noise: symmetric, but not zero, which is the finite-sample floor.  wPLI is near")
    print("the floor for both, because neither has any lag to find.")
Spectral Granger causality, 5-60 Hz, 60 epochs of 2 s (multitaper; mne-connectivity)
                                      simulation   GC 1->2   GC 2->1   coherence     wPLI
 A  x drives y, lag 1 sample, no shared dynamics    0.6394    0.2209      0.9344   0.8951
        B  x drives y, both strongly oscillatory    0.4978    0.0386      0.3749   0.1942
     C  ONE source, zero lag, equal sensor noise    0.2604    0.2769      0.9976   0.1168
   D  ONE source, zero lag, UNEQUAL sensor noise    0.4061    0.0385      0.9870   0.1824

Read row D.  There is ONE generator, there is no lag, and Granger causality reports a strong and
asymmetric directed influence -- from the high-SNR sensor to the low-SNR one -- because unequal
measurement noise makes one channel's past a better predictor than the other's.  Row C shows the same
geometry with equal noise: symmetric, but not zero, which is the finite-sample floor.  wPLI is near
the floor for both, because neither has any lag to find.

8 · The numbers

In [12]:
print("nb-5-2-connectivity -- L5.2 numbers (draft; TODO(confirm) at author review)")
print(f"Data: ds-lemon {SUBJECT}, eyes closed, first {MAX_MINUTES:g} min of the raw recording "
      f"(HTTP Range request, {freed:.0f} MB, deleted)")
print(f"      {len(data_uv)} epochs of {meta['epoch_s']:g} s x {len(ch_names)} EEG channels at {sfreq:g} Hz; "
      f"{L5.licence_line('ds-lemon')}")
print(f"Pipeline: {L5.LEMON_PIPELINE['filter_hz']} Hz FIR, resampled to {L5.LEMON_PIPELINE['resample_hz']:g} Hz, "
      f"{L5.LEMON_PIPELINE['reference']}")
print(f"Band: {BAND[0]:g}-{BAND[1]:g} Hz.  Backend: {info_lib['backend']}")
print()
print(f"{'measure':>8s} {'median':>9s} {'mean':>9s} {'max':>9s} {'pairs above the 95th pct of the surrogate null':>48s}")
for m in METHODS:
    s = L5.matrix_summary(CON[m], m)
    flagged = f"{int((L5.upper_pairs(CON[m]) > thr[m]).sum())} of {coh_v.size} (threshold {thr[m]:.4f})"
    print(f"{m:>8s} {s['median']:9.4f} {s['mean']:9.4f} {s['max']:9.4f} {flagged:>48s}")
print()
print(f"ANSWER KEY -- ex-5-2 (free response): coherence flags {int(flag_coh.sum())} pairs, wPLI flags "
      f"{int(flag_wpli.sum())}, and")
print(f"    {int(only_coh.sum())} pairs are flagged by coherence alone.  The two measures correlate at "
      f"r = {r_cw:+.3f} across pairs.")
print(f"    The largest single disagreement is "
      f"{ch_names[iu[0][order[0]]]}-{ch_names[iu[1][order[0]]]} at {GC_CM[iu[0][order[0]], iu[1][order[0]]]:.1f} cm: "
      f"coherence {coh_v[order[0]]:.4f}, wPLI {wpli_v[order[0]]:.4f}.")
print("    Rubric points: (i) the pair has no measurable lag; (ii) that is what volume conduction produces "
      "and also what")
print("    a true zero-lag interaction would produce, so the two cannot be separated at the sensors; (iii) "
      "the honest")
print("    statement is 'consistent with mixing', not 'not connected'; (iv) wPLI's robustness is bought by "
      "being blind to")
print("    zero-lag coupling entirely.")
print()
print("ANSWER KEY -- reference dependence (same epochs, same band, same estimator):")
for mode in REFS:
    print(f"    {mode:>18s}: median coherence "
          f"{np.median(L5.upper_pairs(ref_con[mode]['coh'])):.4f}, median wPLI "
          f"{np.median(L5.upper_pairs(ref_con[mode]['wpli'])):.4f}")
print("    TP7/TP8 stand in for mastoids; this montage has none.  TODO(confirm).")
print()
print("ANSWER KEY -- removing one linear component (the leading PC, "
      f"{100 * var[0]:.1f} % of variance):")
for m in ("coh", "wpli"):
    b, a_ = L5.upper_pairs(CON[m]), L5.upper_pairs(c_clean[m])
    print(f"    {m:>5s}: median {np.median(b):.4f} -> {np.median(a_):.4f}, largest single change "
          f"{np.abs(a_ - b).max():.4f}")
print()
if gc_rows[0][1] is not None:
    print("ANSWER KEY -- directed measures on simulations where the truth is known:")
    for label, g12, g21, ccoh, cw in gc_rows:
        print(f"    {label:>48s}: GC 1->2 {g12:.4f}, GC 2->1 {g21:.4f}, coherence {ccoh:.4f}, wPLI {cw:.4f}")
    print("    Row D is the warning: one generator, zero lag, unequal sensor noise, and a confident direction.")
print()
print("Pitfalls: pf-volume-conduction-connectivity, pf-reference-changes-everything.")
print(L5.disk_line("disk at the end"))
nb-5-2-connectivity -- L5.2 numbers (draft; TODO(confirm) at author review)
Data: ds-lemon sub-010002, eyes closed, first 3.2 min of the raw recording (HTTP Range request, 60 MB, deleted)
      60 epochs of 2 s x 61 EEG channels at 250 Hz; ds-lemon — LEMON (MPI-Leipzig Mind-Brain-Body): licence CC-BY-4.0, access open (data/directory.yaml)
Pipeline: (1.0, 45.0) Hz FIR, resampled to 250 Hz, average over the 61 EEG channels
Band: 8-12 Hz.  Backend: mne-connectivity 0.9.0 (spectral_connectivity_epochs, mode='fourier')

 measure    median      mean       max   pairs above the 95th pct of the surrogate null
     coh    0.4178    0.4360    0.9955                  1453 of 1830 (threshold 0.2290)
   imcoh    0.0320    0.0447    0.3118                    27 of 1830 (threshold 0.1709)
     plv    0.3418    0.3649    0.9844                  1330 of 1830 (threshold 0.2163)
     pli    0.1074    0.1135    0.2963                     6 of 1830 (threshold 0.2370)
    wpli    0.1851    0.1967    0.6255                    31 of 1830 (threshold 0.3767)

ANSWER KEY -- ex-5-2 (free response): coherence flags 1453 pairs, wPLI flags 31, and
    1422 pairs are flagged by coherence alone.  The two measures correlate at r = +0.226 across pairs.
    The largest single disagreement is P8-P6 at 2.5 cm: coherence 0.9955, wPLI 0.0853.
    Rubric points: (i) the pair has no measurable lag; (ii) that is what volume conduction produces and also what
    a true zero-lag interaction would produce, so the two cannot be separated at the sensors; (iii) the honest
    statement is 'consistent with mixing', not 'not connected'; (iv) wPLI's robustness is bought by being blind to
    zero-lag coupling entirely.

ANSWER KEY -- reference dependence (same epochs, same band, same estimator):
               average: median coherence 0.4327, median wPLI 0.1883
                    Cz: median coherence 0.3329, median wPLI 0.1743
        linked TP7/TP8: median coherence 0.3490, median wPLI 0.1582
    TP7/TP8 stand in for mastoids; this montage has none.  TODO(confirm).

ANSWER KEY -- removing one linear component (the leading PC, 26.8 % of variance):
      coh: median 0.4178 -> 0.3682, largest single change 0.5833
     wpli: median 0.1851 -> 0.1777, largest single change 0.3346

ANSWER KEY -- directed measures on simulations where the truth is known:
     A  x drives y, lag 1 sample, no shared dynamics: GC 1->2 0.6394, GC 2->1 0.2209, coherence 0.9344, wPLI 0.8951
            B  x drives y, both strongly oscillatory: GC 1->2 0.4978, GC 2->1 0.0386, coherence 0.3749, wPLI 0.1942
         C  ONE source, zero lag, equal sensor noise: GC 1->2 0.2604, GC 2->1 0.2769, coherence 0.9976, wPLI 0.1168
       D  ONE source, zero lag, UNEQUAL sensor noise: GC 1->2 0.4061, GC 2->1 0.0385, coherence 0.9870, wPLI 0.1824
    Row D is the warning: one generator, zero lag, unequal sensor noise, and a confident direction.

Pitfalls: pf-volume-conduction-connectivity, pf-reference-changes-everything.
disk at the end: 4.35 GB free on the working volume