Spherical Linear Interpolation and Text-Anchoring for Zero-shot Composed Image Retrieval

TL;DR

This paper introduces a novel zero-shot composed image retrieval method using spherical linear interpolation and text-anchored tuning, achieving state-of-the-art results without relying on expensive annotated datasets.

Contribution

The paper proposes a new zero-shot CIR approach that directly merges image and text representations with Slerp and employs Text-Anchored-Tuning to improve modality alignment.

Findings

Achieves state-of-the-art performance on CIR benchmarks.

TAT improves the effectiveness of Slerp by reducing modality gap.

Method is efficient and serves as a strong initial checkpoint for supervised models.

Abstract

Composed Image Retrieval (CIR) is a complex task that retrieves images using a query, which is configured with an image and a caption that describes desired modifications to that image. Supervised CIR approaches have shown strong performance, but their reliance on expensive manually-annotated datasets restricts their scalability and broader applicability. To address these issues, previous studies have proposed pseudo-word token-based Zero-Shot CIR (ZS-CIR) methods, which utilize a projection module to map images to word tokens. However, we conjecture that this approach has a downside: the projection module distorts the original image representation and confines the resulting composed embeddings to the text-side. In order to resolve this, we introduce a novel ZS-CIR method that uses Spherical Linear Interpolation (Slerp) to directly merge image and text representations by identifying an intermediate embedding of both. Furthermore, we introduce Text-Anchored-Tuning (TAT), a method that fine-tunes the image encoder while keeping the text encoder fixed. TAT closes the modality gap between images and text, making the Slerp process much more effective. Notably, the TAT method is not only efficient in terms of the scale of the training dataset and training time, but it also serves as an excellent initial checkpoint for training supervised CIR models, thereby highlighting its wider potential. The integration of the Slerp-based ZS-CIR with a TAT-tuned model enables our approach to deliver state-of-the-art retrieval performance across CIR benchmarks.