DALR: Dual-level Alignment Learning for Multimodal Sentence Representation Learning

TL;DR

DALR introduces a dual-level alignment approach that improves multimodal sentence representations by addressing cross-modal misalignment and intra-modal divergence, leading to superior performance on semantic tasks.

Contribution

The paper proposes a novel dual-level alignment learning framework that combines fine-grained cross-modal alignment with ranking distillation for better sentence representations.

Findings

Outperforms state-of-the-art baselines on STS and transfer tasks.

Effectively mitigates cross-modal misalignment bias.

Enhances intra-modal semantic consistency.

Abstract

Previous multimodal sentence representation learning methods have achieved impressive performance. However, most approaches focus on aligning images and text at a coarse level, facing two critical challenges:cross-modal misalignment bias and intra-modal semantic divergence, which significantly degrade sentence representation quality. To address these challenges, we propose DALR (Dual-level Alignment Learning for Multimodal Sentence Representation). For cross-modal alignment, we propose a consistency learning module that softens negative samples and utilizes semantic similarity from an auxiliary task to achieve fine-grained cross-modal alignment. Additionally, we contend that sentence relationships go beyond binary positive-negative labels, exhibiting a more intricate ranking structure. To better capture these relationships and enhance representation quality, we integrate ranking distillation with global intra-modal alignment learning. Comprehensive experiments on semantic textual similarity (STS) and transfer (TR) tasks validate the effectiveness of our approach, consistently demonstrating its superiority over state-of-the-art baselines.