Investigating Inner Properties of Multimodal Representation and Semantic Compositionality with Brain-based Componential Semantics

TL;DR

This paper explores the internal properties of multimodal semantic representations by correlating them with brain-based semantics, providing insights into how meaning is encoded and composed across modalities.

Contribution

It introduces interpretation methods linking multimodal representations with brain-based semantics, revealing their properties and the process of semantic compositionality.

Findings

Correlates multimodal representations with brain-based property vectors

Maps distributed vectors to brain-based semantic space

Provides insights into semantic compositionality across modalities

Abstract

Multimodal models have been proven to outperform text-based approaches on learning semantic representations. However, it still remains unclear what properties are encoded in multimodal representations, in what aspects do they outperform the single-modality representations, and what happened in the process of semantic compositionality in different input modalities. Considering that multimodal models are originally motivated by human concept representations, we assume that correlating multimodal representations with brain-based semantics would interpret their inner properties to answer the above questions. To that end, we propose simple interpretation methods based on brain-based componential semantics. First we investigate the inner properties of multimodal representations by correlating them with corresponding brain-based property vectors. Then we map the distributed vector space to the interpretable brain-based componential space to explore the inner properties of semantic compositionality. Ultimately, the present paper sheds light on the fundamental questions of natural language understanding, such as how to represent the meaning of words and how to combine word meanings into larger units.