Siamese Network of Deep Fisher-Vector Descriptors for Image Retrieval

TL;DR

This paper introduces a novel Siamese network combining CNN and Fisher vector components, trained jointly to improve large-scale image retrieval accuracy, outperforming existing methods on standard datasets.

Contribution

It proposes a new Siamese network architecture that jointly learns CNN filters and Fisher vector parameters for enhanced image retrieval.

Findings

Significant improvements over state-of-the-art on Oxford and Paris datasets.

Effective handling of large distractor sets with 1 million images.

Joint learning of CNN and Fisher vector parameters enhances retrieval performance.

Abstract

This paper addresses the problem of large scale image retrieval, with the aim of accurately ranking the similarity of a large number of images to a given query image. To achieve this, we propose a novel Siamese network. This network consists of two computational strands, each comprising of a CNN component followed by a Fisher vector component. The CNN component produces dense, deep convolutional descriptors that are then aggregated by the Fisher Vector method. Crucially, we propose to simultaneously learn both the CNN filter weights and Fisher Vector model parameters. This allows us to account for the evolving distribution of deep descriptors over the course of the learning process. We show that the proposed approach gives significant improvements over the state-of-the-art methods on the Oxford and Paris image retrieval datasets. Additionally, we provide a baseline performance measure for both these datasets with the inclusion of 1 million distractors.