Set-to-Set Hashing with Applications in Visual Recognition

TL;DR

This paper introduces a set-to-set hashing method for efficient large-scale visual retrieval, encoding statistical and structural set information and outperforming previous approaches in accuracy.

Contribution

It presents a novel set representation and a kernel-based hashing framework specifically designed for set-to-set similarity search in visual recognition.

Findings

Outperforms prior methods in visual retrieval tasks

Effective encoding of statistical and structural set information

Scalable to large datasets with improved accuracy

Abstract

Visual data, such as an image or a sequence of video frames, is often naturally represented as a point set. In this paper, we consider the fundamental problem of finding a nearest set from a collection of sets, to a query set. This problem has obvious applications in large-scale visual retrieval and recognition, and also in applied fields beyond computer vision. One challenge stands out in solving the problem---set representation and measure of similarity. Particularly, the query set and the sets in dataset collection can have varying cardinalities. The training collection is large enough such that linear scan is impractical. We propose a simple representation scheme that encodes both statistical and structural information of the sets. The derived representations are integrated in a kernel framework for flexible similarity measurement. For the query set process, we adopt a learning-to-hash pipeline that turns the kernel representations into hash bits based on simple learners, using multiple kernel learning. Experiments on two visual retrieval datasets show unambiguously that our set-to-set hashing framework outperforms prior methods that do not take the set-to-set search setting.