Aligning Brain Signals with Multimodal Speech and Vision Embeddings

TL;DR

This paper investigates how layered multimodal embeddings from wav2vec2 and CLIP models align with brain signals during speech perception, aiming to better understand neural processing of language as a multisensory experience.

Contribution

It compares layer-specific and combined embeddings from wav2vec2 and CLIP models in relation to EEG data, advancing methods for neural decoding of language.

Findings

Multimodal, layer-aware embeddings improve alignment with brain activity.

Combining embeddings from different layers enhances decoding accuracy.

Layer-specific embeddings reveal insights into neural language processing.

Abstract

When we hear the word "house", we don't just process sound, we imagine walls, doors, memories. The brain builds meaning through layers, moving from raw acoustics to rich, multimodal associations. Inspired by this, we build on recent work from Meta that aligned EEG signals with averaged wav2vec2 speech embeddings, and ask a deeper question: which layers of pre-trained models best reflect this layered processing in the brain? We compare embeddings from two models: wav2vec2, which encodes sound into language, and CLIP, which maps words to images. Using EEG recorded during natural speech perception, we evaluate how these embeddings align with brain activity using ridge regression and contrastive decoding. We test three strategies: individual layers, progressive concatenation, and progressive summation. The findings suggest that combining multimodal, layer-aware representations may bring us closer to decoding how the brain understands language, not just as sound, but as experience.