What is Multimodality?

TL;DR

This paper critiques outdated definitions of multimodality in machine learning, proposing a new task-relative framework that emphasizes relevant representations for specific tasks to advance language grounding and natural language understanding.

Contribution

It introduces a novel, task-focused definition of multimodality, addressing foundational gaps and guiding future research in multimodal machine learning.

Findings

Highlights limitations of existing definitions

Proposes a task-relative multimodality framework

Aims to improve language grounding and NLU

Abstract

The last years have shown rapid developments in the field of multimodal machine learning, combining e.g., vision, text or speech. In this position paper we explain how the field uses outdated definitions of multimodality that prove unfit for the machine learning era. We propose a new task-relative definition of (multi)modality in the context of multimodal machine learning that focuses on representations and information that are relevant for a given machine learning task. With our new definition of multimodality we aim to provide a missing foundation for multimodal research, an important component of language grounding and a crucial milestone towards NLU.