Deep Aggregation of Regional Convolutional Activations for Content Based Image Retrieval

TL;DR

This paper introduces a supervised regional aggregation method with learned weights and a new loss function, significantly improving image retrieval performance and training efficiency.

Contribution

It proposes a novel supervised aggregation approach with learned regional weights and a new NRA loss for fine-tuning neural networks in image retrieval.

Findings

Achieves state-of-the-art results on INRIA Holidays dataset.

Provides competitive results on Oxford and Paris datasets.

Reduces training time compared to existing methods.

Abstract

One of the key challenges of deep learning based image retrieval remains in aggregating convolutional activations into one highly representative feature vector. Ideally, this descriptor should encode semantic, spatial and low level information. Even though off-the-shelf pre-trained neural networks can already produce good representations in combination with aggregation methods, appropriate fine tuning for the task of image retrieval has shown to significantly boost retrieval performance. In this paper, we present a simple yet effective supervised aggregation method built on top of existing regional pooling approaches. In addition to the maximum activation of a given region, we calculate regional average activations of extracted feature maps. Subsequently, weights for each of the pooled feature vectors are learned to perform a weighted aggregation to a single feature vector. Furthermore, we apply our newly proposed NRA loss function for deep metric learning to fine tune the backbone neural network and to learn the aggregation weights. Our method achieves state-of-the-art results for the INRIA Holidays data set and competitive results for the Oxford Buildings and Paris data sets while reducing the training time significantly.