Comparing apples to apples in the evaluation of binary coding methods

TL;DR

This paper highlights the importance of proper data normalization and feature mapping for fair evaluation of binary coding methods in nearest neighbor search, revealing that simple methods often outperform complex ones when evaluated correctly.

Contribution

It provides a methodological framework for fair comparison of binary coding methods, emphasizing normalization and feature mapping techniques.

Findings

Proper normalization is crucial for fair comparison.

Simple methods like LSH and ITQ often outperform complex methods.

Evaluation conditions significantly affect method performance results.

Abstract

We discuss methodological issues related to the evaluation of unsupervised binary code construction methods for nearest neighbor search. These issues have been widely ignored in literature. These coding methods attempt to preserve either Euclidean distance or angular (cosine) distance in the binary embedding space. We explain why when comparing a method whose goal is preserving cosine similarity to one designed for preserving Euclidean distance, the original features should be normalized by mapping them to the unit hypersphere before learning the binary mapping functions. To compare a method whose goal is to preserves Euclidean distance to one that preserves cosine similarity, the original feature data must be mapped to a higher dimension by including a bias term in binary mapping functions. These conditions ensure the fair comparison between different binary code methods for the task of nearest neighbor search. Our experiments show under these conditions the very simple methods (e.g. LSH and ITQ) often outperform recent state-of-the-art methods (e.g. MDSH and OK-means).