Generative Editing in the Joint Vision-Language Space for Zero-Shot Composed Image Retrieval

TL;DR

Fusion-Diff introduces a generative editing framework in the joint vision-language space for zero-shot composed image retrieval, effectively bridging modality gaps with high data efficiency and outperforming prior methods.

Contribution

It proposes a novel multimodal fusion editing strategy and a lightweight Control-Adapter for effective zero-shot CIR with limited synthetic data.

Findings

Outperforms prior zero-shot CIR methods on standard benchmarks

Uses only 200K synthetic samples for training

Enhances interpretability through visualization of fused representations

Abstract

Composed Image Retrieval (CIR) enables fine-grained visual search by combining a reference image with a textual modification. While supervised CIR methods achieve high accuracy, their reliance on costly triplet annotations motivates zero-shot solutions. The core challenge in zero-shot CIR (ZS-CIR) stems from a fundamental dilemma: existing text-centric or diffusion-based approaches struggle to effectively bridge the vision-language modality gap. To address this, we propose Fusion-Diff, a novel generative editing framework with high effectiveness and data efficiency designed for multimodal alignment. First, it introduces a multimodal fusion feature editing strategy within a joint vision-language (VL) space, substantially narrowing the modality gap. Second, to maximize data efficiency, the framework incorporates a lightweight Control-Adapter, enabling state-of-the-art performance through fine-tuning on only a limited-scale synthetic dataset of 200K samples. Extensive experiments on standard CIR benchmarks (CIRR, FashionIQ, and CIRCO) demonstrate that Fusion-Diff significantly outperforms prior zero-shot approaches. We further enhance the interpretability of our model by visualizing the fused multimodal representations.