Training Vision Transformers for Image Retrieval

TL;DR

This paper introduces a transformer-based approach for image retrieval that improves upon convolutional methods by using vision transformers trained with a metric learning objective, achieving state-of-the-art results on multiple benchmarks.

Contribution

It is the first to adapt vision transformers for image retrieval with a novel training method combining contrastive loss and entropy regularization, outperforming existing approaches.

Findings

Transformers outperform convolutional models on category-level retrieval benchmarks.

The proposed method achieves state-of-the-art results on Stanford Online Product, In-Shop, and CUB-200.

Transformers are competitive for object retrieval with low-resolution images and short vector representations.

Abstract

Transformers have shown outstanding results for natural language understanding and, more recently, for image classification. We here extend this work and propose a transformer-based approach for image retrieval: we adopt vision transformers for generating image descriptors and train the resulting model with a metric learning objective, which combines a contrastive loss with a differential entropy regularizer. Our results show consistent and significant improvements of transformers over convolution-based approaches. In particular, our method outperforms the state of the art on several public benchmarks for category-level retrieval, namely Stanford Online Product, In-Shop and CUB-200. Furthermore, our experiments on ROxford and RParis also show that, in comparable settings, transformers are competitive for particular object retrieval, especially in the regime of short vector representations and low-resolution images.