Do self-supervised speech and language models extract similar representations as human brain?

TL;DR

This study compares self-supervised speech and language models, Wav2Vec2.0 and GPT-2, revealing they predict similar neural responses in the brain's auditory cortex, emphasizing shared contextual speech representations.

Contribution

It demonstrates that SSL models trained on speech and language tasks share neural correlates, highlighting their convergent speech contextual representations and alignment with brain activity.

Findings

Both models predict auditory cortex responses accurately.

Significant correlation between Wav2Vec2.0 and GPT-2 brain predictions.

Shared speech context explains most variance in neural activity.

Abstract

Speech and language models trained through self-supervised learning (SSL) demonstrate strong alignment with brain activity during speech and language perception. However, given their distinct training modalities, it remains unclear whether they correlate with the same neural aspects. We directly address this question by evaluating the brain prediction performance of two representative SSL models, Wav2Vec2.0 and GPT-2, designed for speech and language tasks. Our findings reveal that both models accurately predict speech responses in the auditory cortex, with a significant correlation between their brain predictions. Notably, shared speech contextual information between Wav2Vec2.0 and GPT-2 accounts for the majority of explained variance in brain activity, surpassing static semantic and lower-level acoustic-phonetic information. These results underscore the convergence of speech contextual representations in SSL models and their alignment with the neural network underlying speech perception, offering valuable insights into both SSL models and the neural basis of speech and language processing.