FreestyleRet: Retrieving Images from Style-Diversified Queries

TL;DR

This paper introduces a new image retrieval task that supports diverse query styles, along with a dataset and a lightweight framework that improves retrieval accuracy by understanding style and texture features.

Contribution

It proposes the first style-diversified retrieval dataset and a novel style-aware retrieval framework utilizing Gram Matrix features and prompt tuning.

Findings

The model outperforms existing retrieval methods on style-diversified queries.

It enables simultaneous retrieval of mixed query styles such as sketch+text or art+text.

Auxiliary information from multiple query styles enhances retrieval performance.

Abstract

Image Retrieval aims to retrieve corresponding images based on a given query. In application scenarios, users intend to express their retrieval intent through various query styles. However, current retrieval tasks predominantly focus on text-query retrieval exploration, leading to limited retrieval query options and potential ambiguity or bias in user intention. In this paper, we propose the Style-Diversified Query-Based Image Retrieval task, which enables retrieval based on various query styles. To facilitate the novel setting, we propose the first Diverse-Style Retrieval dataset, encompassing diverse query styles including text, sketch, low-resolution, and art. We also propose a light-weighted style-diversified retrieval framework. For various query style inputs, we apply the Gram Matrix to extract the query's textural features and cluster them into a style space with style-specific bases. Then we employ the style-init prompt tuning module to enable the visual encoder to comprehend the texture and style information of the query. Experiments demonstrate that our model, employing the style-init prompt tuning strategy, outperforms existing retrieval models on the style-diversified retrieval task. Moreover, style-diversified queries~(sketch+text, art+text, etc) can be simultaneously retrieved in our model. The auxiliary information from other queries enhances the retrieval performance within the respective query.