iSEARLE: Improving Textual Inversion for Zero-Shot Composed Image Retrieval

TL;DR

This paper introduces iSEARLE, a zero-shot approach for composed image retrieval that maps reference images into textual tokens, enabling retrieval without labeled training data, and provides a new benchmark dataset CIRCO.

Contribution

The paper proposes iSEARLE for zero-shot composed image retrieval and introduces CIRCO, a new open-domain dataset with multiple ground truths and semantic categories.

Findings

iSEARLE achieves state-of-the-art results on multiple datasets

CIRCO enables benchmarking of zero-shot CIR methods

The approach generalizes well across different domains and object compositions

Abstract

Given a query consisting of a reference image and a relative caption, Composed Image Retrieval (CIR) aims to retrieve target images visually similar to the reference one while incorporating the changes specified in the relative caption. The reliance of supervised methods on labor-intensive manually labeled datasets hinders their broad applicability. In this work, we introduce a new task, Zero-Shot CIR (ZS-CIR), that addresses CIR without the need for a labeled training dataset. We propose an approach named iSEARLE (improved zero-Shot composEd imAge Retrieval with textuaL invErsion) that involves mapping the visual information of the reference image into a pseudo-word token in CLIP token embedding space and combining it with the relative caption. To foster research on ZS-CIR, we present an open-domain benchmarking dataset named CIRCO (Composed Image Retrieval on Common Objects in context), the first CIR dataset where each query is labeled with multiple ground truths and a semantic categorization. The experimental results illustrate that iSEARLE obtains state-of-the-art performance on three different CIR datasets -- FashionIQ, CIRR, and the proposed CIRCO -- and two additional evaluation settings, namely domain conversion and object composition. The dataset, the code, and the model are publicly available at https://github.com/miccunifi/SEARLE.