MCSE: Multimodal Contrastive Learning of Sentence Embeddings

TL;DR

This paper introduces MCSE, a multimodal contrastive learning method that leverages visual and textual data to produce more semantically meaningful sentence embeddings, outperforming previous models on semantic similarity tasks.

Contribution

The paper presents a novel multimodal contrastive learning approach that enhances sentence embeddings by integrating visual information, improving performance over text-only methods.

Findings

Improves average Spearman's correlation by 1.7% on semantic similarity tasks.

Enhances alignment of semantically similar sentences in embedding space.

Consistently outperforms existing methods across various datasets.

Abstract

Learning semantically meaningful sentence embeddings is an open problem in natural language processing. In this work, we propose a sentence embedding learning approach that exploits both visual and textual information via a multimodal contrastive objective. Through experiments on a variety of semantic textual similarity tasks, we demonstrate that our approach consistently improves the performance across various datasets and pre-trained encoders. In particular, combining a small amount of multimodal data with a large text-only corpus, we improve the state-of-the-art average Spearman's correlation by 1.7%. By analyzing the properties of the textual embedding space, we show that our model excels in aligning semantically similar sentences, providing an explanation for its improved performance.