BrokenBind: Universal Modality Exploration beyond Dataset Boundaries

TL;DR

BrokenBind introduces a universal multi-modal learning framework that can bind any two modalities across different datasets, overcoming dataset limitations and enabling flexible, generalized modality exploration even in low-data scenarios.

Contribution

The paper proposes BrokenBind, a novel method that captures relationships between modalities from different datasets to generate pseudo embeddings, enabling universal modality binding beyond dataset boundaries.

Findings

BrokenBind outperforms existing multi-modal methods in various tasks.

It effectively binds modalities from different datasets despite distribution mismatches.

The approach works well even with limited data in low-data regimes.

Abstract

Multi-modal learning combines various modalities to provide a comprehensive understanding of real-world problems. A common strategy is to directly bind different modalities together in a specific joint embedding space. However, the capability of existing methods is restricted within the modalities presented in the given dataset, thus they are biased when generalizing to unpresented modalities in downstream tasks. As a result, due to such inflexibility, the viability of previous methods is seriously hindered by the cost of acquiring multi-modal datasets. In this paper, we introduce BrokenBind, which focuses on binding modalities that are presented from different datasets. To achieve this, BrokenBind simultaneously leverages multiple datasets containing the modalities of interest and one shared modality. Though the two datasets do not correspond to each other due to distribution mismatch, we can capture their relationship to generate pseudo embeddings to fill in the missing modalities of interest, enabling flexible and generalized multi-modal learning. Under our framework, any two modalities can be bound together, free from the dataset limitation, to achieve universal modality exploration. Further, to reveal the capability of our method, we study intensified scenarios where more than two datasets are needed for modality binding and show the effectiveness of BrokenBind in low-data regimes. Through extensive evaluation, we carefully justify the superiority of BrokenBind compared to well-known multi-modal baseline methods.