Connecting Compression Spaces with Transformer for Approximate Nearest Neighbor Search

TL;DR

This paper introduces a transformer-based feature compression method for Approximate Nearest Neighbor Search that enhances efficiency while maintaining or improving search accuracy across various ANNS techniques.

Contribution

It presents a novel transformer-based network with an INRP loss function for effective feature compression in ANNS, improving speed and accuracy.

Findings

Significantly improves efficiency of ANNS methods

Preserves or enhances search accuracy after compression

Applicable across multiple ANNS algorithms

Abstract

We propose a generic feature compression method for Approximate Nearest Neighbor Search (ANNS) problems, which speeds up existing ANNS methods in a plug-and-play manner. Specifically, based on transformer, we propose a new network structure to compress the feature into a low dimensional space, and an inhomogeneous neighborhood relationship preserving (INRP) loss that aims to maintain high search accuracy. Specifically, we use multiple compression projections to cast the feature into many low dimensional spaces, and then use transformer to globally optimize these projections such that the features are well compressed following the guidance from our loss function. The loss function is designed to assign high weights on point pairs that are close in original feature space, and keep their distances in projected space. Keeping these distances helps maintain the eventual top-k retrieval accuracy, and down weighting others creates room for feature compression. In experiments, we run our compression method on public datasets, and use the compressed features in graph based, product quantization and scalar quantization based ANNS solutions. Experimental results show that our compression method can significantly improve the efficiency of these methods while preserves or even improves search accuracy, suggesting its broad potential impact on real world applications.