Investigating the Vision Transformer Model for Image Retrieval Tasks

TL;DR

This paper demonstrates that Vision Transformers can serve as effective, training-free global descriptors for image retrieval, offering a competitive alternative to traditional CNN-based methods without the need for fine-tuning.

Contribution

It introduces a plug-and-play Vision Transformer-based descriptor for image retrieval that does not require training, simplifying the process and providing a new baseline.

Findings

Achieves competitive results on multiple datasets.

Outperforms many existing handcrafted and CNN-based descriptors.

Requires no training or fine-tuning.

Abstract

This paper introduces a plug-and-play descriptor that can be effectively adopted for image retrieval tasks without prior initialization or preparation. The description method utilizes the recently proposed Vision Transformer network while it does not require any training data to adjust parameters. In image retrieval tasks, the use of Handcrafted global and local descriptors has been very successfully replaced, over the last years, by the Convolutional Neural Networks (CNN)-based methods. However, the experimental evaluation conducted in this paper on several benchmarking datasets against 36 state-of-the-art descriptors from the literature demonstrates that a neural network that contains no convolutional layer, such as Vision Transformer, can shape a global descriptor and achieve competitive results. As fine-tuning is not required, the presented methodology's low complexity encourages adoption of the architecture as an image retrieval baseline model, replacing the traditional and well adopted CNN-based approaches and inaugurating a new era in image retrieval approaches.