Low-dimensional Embodied Semantics for Music and Language

TL;DR

This paper introduces low-dimensional vector embeddings derived from joint modeling of multiple human brains' neural responses, capturing shared semantics in music and language more effectively than high-dimensional data.

Contribution

It proposes a novel method for representing shared semantics through low-dimensional embeddings learned from multiple subjects' neural data, improving classification tasks.

Findings

Low-dimensional embeddings outperform original voxel spaces in classification.

Joint modeling increases semantic richness of the representations.

Embeddings effectively capture shared semantics across subjects.

Abstract

Embodied cognition states that semantics is encoded in the brain as firing patterns of neural circuits, which are learned according to the statistical structure of human multimodal experience. However, each human brain is idiosyncratically biased, according to its subjective experience history, making this biological semantic machinery noisy with respect to the overall semantics inherent to media artifacts, such as music and language excerpts. We propose to represent shared semantics using low-dimensional vector embeddings by jointly modeling several brains from human subjects. We show these unsupervised efficient representations outperform the original high-dimensional fMRI voxel spaces in proxy music genre and language topic classification tasks. We further show that joint modeling of several subjects increases the semantic richness of the learned latent vector spaces.