Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval

TL;DR

This paper introduces OSrCIR, a training-free, one-stage reasoning method using multimodal large language models for zero-shot composed image retrieval, improving accuracy by retaining visual details and enhancing interpretability.

Contribution

The paper presents a novel one-stage, training-free Reflective Chain-of-Thought framework that outperforms existing two-stage methods in zero-shot composed image retrieval.

Findings

Achieves 1.80% to 6.44% performance improvements over existing methods.

Sets new state-of-the-art results in zero-shot composed image retrieval.

Enhances interpretability through reflective reasoning aligning manipulation intent with reference image cues.

Abstract

Composed Image Retrieval (CIR) aims to retrieve target images that closely resemble a reference image while integrating user-specified textual modifications, thereby capturing user intent more precisely. Existing training-free zero-shot CIR (ZS-CIR) methods often employ a two-stage process: they first generate a caption for the reference image and then use Large Language Models for reasoning to obtain a target description. However, these methods suffer from missing critical visual details and limited reasoning capabilities, leading to suboptimal retrieval performance. To address these challenges, we propose a novel, training-free one-stage method, One-Stage Reflective Chain-of-Thought Reasoning for ZS-CIR (OSrCIR), which employs Multimodal Large Language Models to retain essential visual information in a single-stage reasoning process, eliminating the information loss seen in two-stage methods. Our Reflective Chain-of-Thought framework further improves interpretative accuracy by aligning manipulation intent with contextual cues from reference images. OSrCIR achieves performance gains of 1.80% to 6.44% over existing training-free methods across multiple tasks, setting new state-of-the-art results in ZS-CIR and enhancing its utility in vision-language applications. Our code will be available at https://github.com/Pter61/osrcir2024/.