DECAF: Dynamic Envelope Context-Aware Fusion for Speech-Envelope Reconstruction from EEG

TL;DR

This paper introduces a dynamic, state-space fusion model for speech envelope reconstruction from EEG that leverages temporal speech context, significantly improving accuracy over static methods and advancing neural decoding techniques.

Contribution

It presents a novel dynamic framework using a state-space model with a learned gating mechanism to fuse neural estimates and speech context for improved envelope reconstruction.

Findings

Significant performance improvements over static EEG-only baselines.

Effective utilization of temporal speech context enhances reconstruction fidelity.

Reframes envelope reconstruction as a dynamic state-estimation problem.

Abstract

Reconstructing the speech audio envelope from scalp neural recordings (EEG) is a central task for decoding a listener's attentional focus in applications like neuro-steered hearing aids. Current methods for this reconstruction, however, face challenges with fidelity and noise. Prevailing approaches treat it as a static regression problem, processing each EEG window in isolation and ignoring the rich temporal structure inherent in continuous speech. This study introduces a new, dynamic framework for envelope reconstruction that leverages this structure as a predictive temporal prior. We propose a state-space fusion model that combines direct neural estimates from EEG with predictions from recent speech context, using a learned gating mechanism to adaptively balance these cues. To validate this approach, we evaluate our model on the ICASSP 2023 Stimulus Reconstruction benchmark demonstrating significant improvements over static, EEG-only baselines. Our analyses reveal a powerful synergy between the neural and temporal information streams. Ultimately, this work reframes envelope reconstruction not as a simple mapping, but as a dynamic state-estimation problem, opening a new direction for developing more accurate and coherent neural decoding systems.