Neural Codes for Image Retrieval

TL;DR

This paper explores the use of neural activations from CNNs as high-level image descriptors for retrieval, demonstrating their effectiveness, robustness to compression, and improvements with discriminative dimensionality reduction.

Contribution

It introduces neural codes as effective image descriptors for retrieval, evaluates their performance across benchmarks, and proposes compression and dimensionality reduction techniques to enhance their utility.

Findings

Neural codes perform competitively even when CNNs are trained on unrelated tasks.

PCA compression yields compact codes with state-of-the-art accuracy.

Discriminative dimensionality reduction further improves retrieval performance.

Abstract

It has been shown that the activations invoked by an image within the top layers of a large convolutional neural network provide a high-level descriptor of the visual content of the image. In this paper, we investigate the use of such descriptors (neural codes) within the image retrieval application. In the experiments with several standard retrieval benchmarks, we establish that neural codes perform competitively even when the convolutional neural network has been trained for an unrelated classification task (e.g.\ Image-Net). We also evaluate the improvement in the retrieval performance of neural codes, when the network is retrained on a dataset of images that are similar to images encountered at test time. We further evaluate the performance of the compressed neural codes and show that a simple PCA compression provides very good short codes that give state-of-the-art accuracy on a number of datasets. In general, neural codes turn out to be much more resilient to such compression in comparison other state-of-the-art descriptors. Finally, we show that discriminative dimensionality reduction trained on a dataset of pairs of matched photographs improves the performance of PCA-compressed neural codes even further. Overall, our quantitative experiments demonstrate the promise of neural codes as visual descriptors for image retrieval.