Cross-Modal Attention Alignment Network with Auxiliary Text Description for zero-shot sketch-based image retrieval

TL;DR

This paper introduces a novel zero-shot sketch-based image retrieval method leveraging large-scale language models to generate auxiliary text descriptions, enabling better cross-modal alignment and improved retrieval performance.

Contribution

The approach uniquely integrates LLM-generated textual descriptions with visual data, enhancing zero-shot generalization in sketch-based image retrieval.

Findings

Outperforms state-of-the-art ZS-SBIR methods on three benchmarks.

Effectively leverages LLM-generated descriptions for cross-modal alignment.

Demonstrates superior zero-shot retrieval accuracy.

Abstract

In this paper, we study the problem of zero-shot sketch-based image retrieval (ZS-SBIR). The prior methods tackle the problem in a two-modality setting with only category labels or even no textual information involved. However, the growing prevalence of Large-scale pre-trained Language Models (LLMs), which have demonstrated great knowledge learned from web-scale data, can provide us with an opportunity to conclude collective textual information. Our key innovation lies in the usage of text data as auxiliary information for images, thus leveraging the inherent zero-shot generalization ability that language offers. To this end, we propose an approach called Cross-Modal Attention Alignment Network with Auxiliary Text Description for zero-shot sketch-based image retrieval. The network consists of three components: (i) a Description Generation Module that generates textual descriptions for each training category by prompting an LLM with several interrogative sentences, (ii) a Feature Extraction Module that includes two ViTs for sketch and image data, a transformer for extracting tokens of sentences of each training category, finally (iii) a Cross-modal Alignment Module that exchanges the token features of both text-sketch and text-image using cross-attention mechanism, and align the tokens locally and globally. Extensive experiments on three benchmark datasets show our superior performances over the state-of-the-art ZS-SBIR methods.