MERIT: Multilingual Semantic Retrieval with Interleaved Multi-Condition Query

TL;DR

This paper introduces MERIT, a large multilingual dataset for interleaved multi-condition semantic retrieval, and proposes Coral, a fine-tuning framework that significantly improves retrieval performance by capturing both global and conditional information.

Contribution

The paper presents the first multilingual dataset for interleaved multi-condition retrieval and introduces Coral, a novel fine-tuning method that enhances model understanding of complex queries.

Findings

Coral outperforms traditional models by 45.9% on MERIT.

Existing models mainly focus on global semantics, neglecting conditional details.

Coral generalizes well across 8 retrieval benchmarks.

Abstract

Semantic retrieval is crucial for modern applications yet remains underexplored in current research. Existing datasets are limited to single languages, single images, or singular retrieval conditions, often failing to fully exploit the expressive capacity of visual information as evidenced by maintained performance when images are replaced with captions. However, practical retrieval scenarios frequently involve interleaved multi-condition queries with multiple images. Hence, this paper introduces MERIT, the first multilingual dataset for interleaved multi-condition semantic retrieval, comprising 320,000 queries with 135,000 products in 5 languages, covering 7 distinct product categories. Extensive experiments on MERIT identify existing models's limitation: focusing solely on global semantic information while neglecting specific conditional elements in queries. Consequently, we propose Coral, a novel fine-tuning framework that adapts pre-trained MLLMs by integrating embedding reconstruction to preserve fine-grained conditional elements and contrastive learning to extract comprehensive global semantics. Experiments demonstrate that Coral achieves a 45.9% performance improvement over conventional approaches on MERIT, with strong generalization capabilities validated across 8 established retrieval benchmarks. Collectively, our contributions - a novel dataset, identification of critical limitations in existing approaches, and an innovative fine-tuning framework - establish a foundation for future research in interleaved multi-condition semantic retrieval.