MCoT-MVS: Multi-level Vision Selection by Multi-modal Chain-of-Thought Reasoning for Composed Image Retrieval

TL;DR

This paper introduces MCoT-MVS, a novel multi-level vision selection method guided by multi-modal reasoning, significantly improving composed image retrieval accuracy by effectively extracting and fusing visual and textual cues.

Contribution

The paper proposes a multi-modal chain-of-thought reasoning framework that enhances visual cue selection and fusion for improved composed image retrieval performance.

Findings

Achieves state-of-the-art results on CIRR and FashionIQ benchmarks.

Effectively extracts discriminative visual features guided by reasoning cues.

Outperforms existing methods consistently across multiple metrics.

Abstract

Composed Image Retrieval (CIR) aims to retrieve target images based on a reference image and modified texts. However, existing methods often struggle to extract the correct semantic cues from the reference image that best reflect the user's intent under textual modification prompts, resulting in interference from irrelevant visual noise. In this paper, we propose a novel Multi-level Vision Selection by Multi-modal Chain-of-Thought Reasoning (MCoT-MVS) for CIR, integrating attention-aware multi-level vision features guided by reasoning cues from a multi-modal large language model (MLLM). Specifically, we leverage an MLLM to perform chain-of-thought reasoning on the multimodal composed input, generating the retained, removed, and target-inferred texts. These textual cues subsequently guide two reference visual attention selection modules to selectively extract discriminative patch-level and instance-level semantics from the reference image. Finally, to effectively fuse these multi-granular visual cues with the modified text and the imagined target description, we design a weighted hierarchical combination module to align the composed query with target images in a unified embedding space. Extensive experiments on two CIR benchmarks, namely CIRR and FashionIQ, demonstrate that our approach consistently outperforms existing methods and achieves new state-of-the-art performance. Code and trained models are publicly released.