TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrieval

TL;DR

TEMA introduces a novel framework and datasets for multi-modification composed image retrieval, addressing limitations of previous methods and improving accuracy and efficiency in practical scenarios.

Contribution

The paper presents TEMA, the first CIR framework designed for multi-modification, along with new instruction-rich datasets, enhancing real-world applicability.

Findings

TEMA outperforms existing methods on four benchmark datasets.

The datasets enable more comprehensive evaluation of CIR models.

TEMA balances retrieval accuracy with computational efficiency.

Abstract

Composed Image Retrieval (CIR) is an important image retrieval paradigm that enables users to retrieve a target image using a multimodal query that consists of a reference image and modification text. Although research on CIR has made significant progress, prevailing setups still rely simple modification texts that typically cover only a limited range of salient changes, which induces two limitations highly relevant to practical applications, namely Insufficient Entity Coverage and Clause-Entity Misalignment. In order to address these issues and bring CIR closer to real-world use, we construct two instruction-rich multi-modification datasets, M-FashionIQ and M-CIRR. In addition, we propose TEMA, the Text-oriented Entity Mapping Architecture, which is the first CIR framework designed for multi-modification while also accommodating simple modifications. Extensive experiments on four benchmark datasets demonstrate that TEMA's superiority in both original and multi-modification scenarios, while maintaining an optimal balance between retrieval accuracy and computational efficiency. Our codes and constructed multi-modification dataset (M-FashionIQ and M-CIRR) are available at https://github.com/lee-zixu/ACL26-TEMA/.