AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Brain-Speech Decoder Opens a Window Into Its Own Decisions

Magnetoencephalography (MEG) can record the brain’s magnetic fields as a person listens to a story. Deep networks trained on such recordings can retrieve the matching three-second audio clip from more than a thousand candidates. But a high score does not reveal what the network detected, or where in the brain that information comes from. A new study describes a speech decoder built so that its internal weights can be read as cortical locations and rhythms, and it uses that design to ask which properties of speech actually drive retrieval.

The model builds on earlier MEG-to-audio retrieval work. It learns a contrastive objective: MEG and audio embeddings from the same segment should be similar, using wav2vec 2.0 audio features as targets. The new front end is physically grounded. Spatial attention is parameterized with spherical harmonics, the natural basis for the roughly spherical MEG sensor helmet. The subject-specific representation is reduced to 25 branches, each with its own spatial filter and a 150-millisecond temporal filter. Ocular and cardiac components are removed before training, so the model cannot lean on eye or heartbeat artifacts. Because spatial and temporal filtering are factorized, trained weights can be projected back onto a cortical surface with standard inverse methods.

The researchers also intervened on the input. For 19 candidate speech features, they replaced MEG intervals associated with that feature with matched donor MEG from either feature-present or feature-absent intervals; if feature-absent substitution hurts retrieval more than the matched feature-present control, the decoder is using that feature. The largest effect was silence: swapping out MEG recorded during silences worsened the correct clip’s rank by 75.6 positions relative to the control condition, and the effect appeared in all 27 participants. High loudness, vowels and strong acoustic onsets also produced large effects. Fifteen of 19 features reached significance after correction for multiple comparisons. Random word lists behaved differently: substituting coherent-narrative MEG into word-list intervals improved retrieval by about 17.8 rank positions. The authors interpret this as evidence that narrative structure supports the neural tracking on which retrieval depends.

The main model, trained on the MEG-MASC dataset, reached 39.75% ± 0.34% Top-1 accuracy—the correct clip ranked first—and 70.40% ± 0.31% Top-10 accuracy among 1,005 candidates across six trained solutions, with about 486,000 trainable parameters—roughly 20 times fewer than the earlier brain decoder of Défossez et al. Those accuracy scores are not directly comparable, the researchers caution, because evaluation protocols and preprocessing differ. Within their own fixed pipeline, a 270-branch, five-block variant with 14.8 times more parameters scored 3.60 percentage points lower on Top-1. Mapped sources included bilateral auditory cortex, superior temporal gyrus and frontal regions, and left-lateralized branches carried an additional faster rhythm near 13.3 Hz.

The study has clear limits. It uses one corpus with 27 participants and fixed narratives; source mapping relied on a common template because defacing truncated most individual MRI scans. Feature effects overlap with one another and have different mask durations, so their ordering is not a calibrated measure of encoding strength. The interpretability is also strongest in the first layer, leaving more complex space-time dynamics unresolved. Still, the work shows that an inspectable decoder can stay in the same performance range as opaque ones, turning a retrieval benchmark into a tool for probing how natural speech is represented in the human brain—without claiming the model reads words one by one.