Efficient Querying from Weighted Binary Codes

TL;DR

This paper introduces an efficient method for querying weighted binary codes using multi-index hash tables, significantly improving search speed and accuracy for large-scale datasets with billions of codes.

Contribution

It proposes novel algorithms for fast, non-exhaustive ranking of weighted binary codes, addressing the open issue of efficient querying.

Findings

Achieves over 1000x speedup compared to linear scan on large datasets.

Maintains high search accuracy with the proposed algorithms.

Validated on three large-scale datasets with millions to billions of codes.

Abstract

Binary codes are widely used to represent the data due to their small storage and efficient computation. However, there exists an ambiguity problem that lots of binary codes share the same Hamming distance to a query. To alleviate the ambiguity problem, weighted binary codes assign different weights to each bit of binary codes and compare the binary codes by the weighted Hamming distance. Till now, performing the querying from the weighted binary codes efficiently is still an open issue. In this paper, we propose a new method to rank the weighted binary codes and return the nearest weighted binary codes of the query efficiently. In our method, based on the multi-index hash tables, two algorithms, the table bucket finding algorithm and the table merging algorithm, are proposed to select the nearest weighted binary codes of the query in a non-exhaustive and accurate way. The proposed algorithms are justified by proving their theoretic properties. The experiments on three large-scale datasets validate both the search efficiency and the search accuracy of our method. Especially for the number of weighted binary codes up to one billion, our method shows a great improvement of more than 1000 times faster than the linear scan.