Better audio representations are more brain-like: linking model-brain alignment with performance in downstream auditory tasks

TL;DR

This study shows that advanced self-supervised audio models with strong downstream task performance also have internal representations more similar to brain activity, and this similarity emerges early during pretraining.

Contribution

It demonstrates a link between model-brain alignment and task performance in audio models, revealing that brain-like representations emerge naturally during training.

Findings

Self-supervised models better predict auditory cortex activity.

Model performance correlates with brain alignment (r > 0.8).

Brain similarity increases early during pretraining.

Abstract

Artificial neural networks are increasingly powerful models of brain computation, yet it remains unclear whether improving their performance in downstream tasks also makes their internal representations more similar to brain signals. To address this question in the auditory domain, we quantified the alignment between the internal representations of 36 different audio models and brain activity from two independent fMRI datasets. Using voxel-wise and component-wise regression, and representation similarity analysis, we found that recent self-supervised audio models with strong performance in diverse downstream tasks are better predictors of auditory cortex activity than previously studied models. To assess the quality of the audio representations, we evaluated these models in 6 auditory tasks from the HEAREval benchmark, spanning music, speech, and environmental sounds. This revealed strong positive Pearson correlations (r > 0.8) between a model's overall task performance and its alignment with brain representations. Finally, we analyzed the evolution of the similarity between audio and brain representations during the pretraining of EnCodecMAE, a recent audio representation model. We discovered that brain similarity increases progressively and emerges early during pretraining, despite the model not being explicitly optimized for this objective. This suggests that brain-like representations can be an emergent byproduct of learning to reconstruct missing information from naturalistic audio data.