MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrieval

TL;DR

MIRe introduces a novel multimodal retrieval framework that enhances query understanding by enabling modality interaction without fusing features, effectively addressing the text-dominant issue and improving zero-shot retrieval performance.

Contribution

The paper proposes a fusion-free modality interaction approach and a new pre-training dataset, advancing multimodal retrieval by mitigating text dominance and improving query comprehension.

Findings

Significant performance gains on four benchmarks under zero-shot settings.

Pre-training with extended question-answer passages improves multimodal query understanding.

Ablation studies confirm the effectiveness of the modality interaction strategy.

Abstract

Recent multimodal retrieval methods have endowed text-based retrievers with multimodal capabilities by utilizing pre-training strategies for visual-text alignment. They often directly fuse the two modalities for cross-reference during the alignment to understand multimodal queries. However, existing methods often overlook crucial visual information due to a text-dominant issue, which overly depends on text-driven signals. In this paper, we introduce MIRe, a retrieval framework that achieves modality interaction without fusing textual features during the alignment. Our method allows the textual query to attend to visual embeddings while not feeding text-driven signals back into the visual representations. Additionally, we construct a pre-training dataset for multimodal query retrieval by transforming concise question-answer pairs into extended passages. Our experiments demonstrate that our pre-training strategy significantly enhances the understanding of multimodal queries, resulting in strong performance across four multimodal retrieval benchmarks under zero-shot settings. Moreover, our ablation studies and analyses explicitly verify the effectiveness of our framework in mitigating the text-dominant issue. Our code is publicly available: https://github.com/yeongjoonJu/MIRe