Compressing then Matching: An Efficient Pre-training Paradigm for Multimodal Embedding

TL;DR

This paper introduces CoMa, a pre-training paradigm that efficiently enhances multimodal language models' embedding capabilities, leading to state-of-the-art results with less data and computational effort.

Contribution

Proposes CoMa, a novel compressed pre-training stage that decouples understanding and contrastive learning, improving efficiency and effectiveness of multimodal embeddings.

Findings

CoMa achieves state-of-the-art results on MMEB with limited data.

Transforming MLLMs into competitive embedding models with minimal pre-training.

Improved efficiency and effectiveness in multimodal embedding learning.

Abstract

Multimodal Large Language Models advance multimodal representation learning by acquiring transferable semantic embeddings, thereby substantially enhancing performance across a range of vision-language tasks, including cross-modal retrieval, clustering, and classification. An effective embedding is expected to comprehensively preserve the semantic content of the input while simultaneously emphasizing features that are discriminative for downstream tasks. Recent approaches demonstrate that MLLMs can be adapted into competitive embedding models via large-scale contrastive learning, enabling the simultaneous optimization of two complementary objectives. We argue that the two aforementioned objectives can be decoupled: a comprehensive understanding of the input enables the embedding model to achieve superior performance on downstream tasks via contrastive learning. In this paper, we propose CoMa, a compressed pre-training phase, which serves as a warm-up stage for contrastive learning. Experiments demonstrate that with only a small amount of pre-training data, we can transform an MLLM into a competitive embedding model. CoMa achieves new state-of-the-art results among MLLMs of comparable size on the MMEB, realizing optimization in both efficiency and effectiveness. Our project is available at https://github.com/Trustworthy-Information-Access/CoMa.