Distance Comparison Operations Are Not Silver Bullets in Vector Similarity Search: A Benchmark Study on Their Merits and Limits

TL;DR

This benchmark study evaluates the performance and limitations of Distance Comparison Operations in vector similarity search across various datasets and hardware, revealing their current unsuitability for production systems.

Contribution

The paper provides a comprehensive benchmark analysis of 8 DCO algorithms, highlighting their sensitivities and potential benefits in index construction and data updates.

Findings

DCO methods are highly sensitive to data dimensionality.

Performance degrades with out-of-distribution queries.

They can sometimes accelerate index construction and updates.

Abstract

Distance Comparison Operations (DCOs), which decide whether the distance between a data vector and a query is within a threshold, are a critical performance bottleneck in vector similarity search. Recent DCO methods that avoid full-dimensional distance computations promise significant speedups, but their readiness for production vector database systems remains an open question. To address this, we conduct a comprehensive benchmark of 8 DCO algorithms across 10 datasets (with up to 100M vectors and 12,288 dimensions) and diverse hardware configurations (CPUs with/without SIMD, and GPUs). Our study reveals that these methods are not silver bullets: their efficiency is highly sensitive to data dimensionality, degrades under out-of-distribution queries, and is unstable across hardware. Yet, our evaluation also demonstrates often-overlooked merits: they can accelerate index construction and data updates. Despite these benefits, their unstable performance, which can be slower than a full-dimensional scan, leads us to conclude that recent algorithmic advancements in DCO are not yet ready for production deployment.