With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You

TL;DR

This paper introduces STRUCTURE, a regularization technique that enables effective multimodal alignment with limited paired data by preserving neighborhood geometry and aligning layers with high representational similarity, significantly improving zero-shot tasks.

Contribution

The paper proposes a novel regularization method, STRUCTURE, for aligning pretrained unimodal models with limited data, and demonstrates its effectiveness across multiple benchmarks.

Findings

High-quality alignment achievable with less than 1% of typical data

Aligning layers with highest representational similarity improves performance

Substantial gains in zero-shot classification and retrieval benchmarks

Abstract

Multimodal models have demonstrated powerful capabilities in complex tasks requiring multimodal alignment, including zero-shot classification and cross-modal retrieval. However, existing models typically rely on millions of paired multimodal samples, which are prohibitively expensive or infeasible to obtain in many domains. In this work, we explore the feasibility of building multimodal models with limited amount of paired data by aligning pretrained unimodal foundation models. We show that high-quality alignment is possible with as few as tens of thousands of paired samples$\unicode{x2013}$less than $1\%$ of the data typically used in the field. To achieve this, we introduce STRUCTURE, an effective regularization technique that preserves the neighborhood geometry of the latent space of unimodal encoders. Additionally, we show that aligning last layers is often suboptimal and demonstrate the benefits of aligning the layers with the highest representational similarity across modalities. These two components can be readily incorporated into existing alignment methods, yielding substantial gains across 24 zero-shot image classification and retrieval benchmarks, with average relative improvement of $51.6\%$ in classification and $91.8\%$ in retrieval tasks. Our results highlight the effectiveness and broad applicability of our framework for limited-sample multimodal learning and offer a promising path forward for resource-constrained domains.