ReCALL: Recalibrating Capability Degradation for MLLM-based Composed Image Retrieval

TL;DR

ReCALL is a framework that improves composed image retrieval by diagnosing and correcting capability degradation in vision-language models, leading to state-of-the-art results.

Contribution

It introduces a diagnose-generate-refine pipeline to recalibrate MLLM-based retrievers, addressing their capability degradation in compositional reasoning.

Findings

ReCALL achieves state-of-the-art performance on CIRR and FashionIQ datasets.

The framework effectively recalibrates degraded capabilities of MLLM-based retrievers.

Extensive experiments validate the robustness and effectiveness of ReCALL.

Abstract

Composed Image Retrieval (CIR) aims to retrieve target images based on a hybrid query comprising a reference image and a modification text. Early dual-tower Vision-Language Models (VLMs) struggle with cross-modality compositional reasoning required for this task. While adapting generative Multimodal Large Language Models (MLLMs) for retrieval offers a promising direction, we identify that this strategy overlooks a fundamental issue: compressing a generative MLLM into a single-embedding discriminative retriever triggers a paradigm conflict, which leads to Capability Degradation - the deterioration of native fine-grained reasoning after retrieval adaptation. To address this challenge, we propose ReCALL, a model-agnostic framework that follows a diagnose-generate-refine pipeline: First, we diagnose cognitive blind spots of the retriever via self-guided informative instance mining. Next, we generate corrective instructions and triplets by prompting the foundation MLLM and conduct quality control with VQA-based consistency filtering. Finally, we refine the retriever through continual training on these triplets with a grouped contrastive scheme, thereby internalizing fine-grained visual-semantic distinctions and realigning the discriminative embedding space of retriever with intrinsic compositional reasoning within the MLLM. Extensive experiments on CIRR and FashionIQ show that ReCALL consistently recalibrates degraded capabilities and achieves state-of-the-art performance. Code is available at https://github.com/RemRico/Recall.