Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning

TL;DR

This paper argues that multiplicity, or many-to-many relationships across modalities, is an inherent challenge in multimodal learning that affects data, training, and evaluation, requiring new research approaches.

Contribution

It highlights multiplicity as a fundamental issue in multimodal learning and advocates for developing multiplicity-aware frameworks and dataset protocols.

Findings

Multiplicity causes training uncertainty.

It leads to unreliable evaluation.

It indicates low dataset quality.

Abstract

Multimodal learning has seen remarkable progress, particularly with the emergence of large-scale pre-training across various modalities. However, most current approaches are built on the assumption of a deterministic, one-to-one alignment between modalities. This oversimplifies real-world multimodal relationships, where their nature is inherently many-to-many. This phenomenon, named multiplicity, is not a side-effect of noise or annotation error, but an inevitable outcome of semantic abstraction, representational asymmetry, and task-dependent ambiguity in multimodal tasks. This position paper argues that multiplicity is a fundamental bottleneck that manifests across all stages of the multimodal learning pipeline: from data construction to training and evaluation. This paper examines the causes and consequences of multiplicity, and highlights how multiplicity introduces training uncertainty, unreliable evaluation, and low dataset quality. This position calls for new research directions on multimodal learning: novel multiplicity-aware learning frameworks and dataset construction protocols considering multiplicity.