Interferences in match kernels

TL;DR

This paper addresses interference in match kernels used for image representation, proposing two weighting strategies to improve descriptor contribution balance, leading to enhanced image search performance.

Contribution

It introduces two novel methods for weighting local descriptors in match kernels, improving image retrieval accuracy over existing approaches.

Findings

Significant performance improvements on public image retrieval benchmarks.

Effective equalization of descriptor contributions enhances similarity measures.

Methods outperform state-of-the-art in short and mid-size vector image search.

Abstract

We consider the design of an image representation that embeds and aggregates a set of local descriptors into a single vector. Popular representations of this kind include the bag-of-visual-words, the Fisher vector and the VLAD. When two such image representations are compared with the dot-product, the image-to-image similarity can be interpreted as a match kernel. In match kernels, one has to deal with interference, i.e. with the fact that even if two descriptors are unrelated, their matching score may contribute to the overall similarity. We formalise this problem and propose two related solutions, both aimed at equalising the individual contributions of the local descriptors in the final representation. These methods modify the aggregation stage by including a set of per-descriptor weights. They differ by the objective function that is optimised to compute those weights. The first is a "democratisation" strategy that aims at equalising the relative importance of each descriptor in the set comparison metric. The second one involves equalising the match of a single descriptor to the aggregated vector. These concurrent methods give a substantial performance boost over the state of the art in image search with short or mid-size vectors, as demonstrated by our experiments on standard public image retrieval benchmarks.