Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities

TL;DR

Symile introduces a simple, model-agnostic contrastive learning method that captures higher-order joint information across multiple modalities, outperforming pairwise approaches like CLIP in diverse tasks.

Contribution

The paper presents Symile, a novel contrastive learning framework that effectively models higher-order relationships among multiple modalities, addressing limitations of pairwise methods.

Findings

Symile outperforms CLIP in cross-modal classification and retrieval.

Symile effectively handles missing modalities during training and inference.

The approach is validated on diverse datasets, including multilingual and clinical data.

Abstract

Contrastive learning methods, such as CLIP, leverage naturally paired data-for example, images and their corresponding text captions-to learn general representations that transfer efficiently to downstream tasks. While such approaches are generally applied to two modalities, domains such as robotics, healthcare, and video need to support many types of data at once. We show that the pairwise application of CLIP fails to capture joint information between modalities, thereby limiting the quality of the learned representations. To address this issue, we present Symile, a simple contrastive learning approach that captures higher-order information between any number of modalities. Symile provides a flexible, architecture-agnostic objective for learning modality-specific representations. To develop Symile's objective, we derive a lower bound on total correlation, and show that Symile representations for any set of modalities form a sufficient statistic for predicting the remaining modalities. Symile outperforms pairwise CLIP, even with modalities missing in the data, on cross-modal classification and retrieval across several experiments including on an original multilingual dataset of 33M image, text and audio samples and a clinical dataset of chest X-rays, electrocardiograms, and laboratory measurements. All datasets and code used in this work are publicly available at https://github.com/rajesh-lab/symile.