TRIM: Accelerating High-Dimensional Vector Similarity Search with Enhanced Triangle-Inequality-Based Pruning

TL;DR

TRIM significantly improves high-dimensional vector similarity search efficiency by enhancing triangle-inequality-based pruning, reducing data access, and integrating seamlessly with existing methods, leading to substantial speedups and lower I/O costs.

Contribution

The paper introduces TRIM, a novel approach that enhances triangle-inequality-based pruning in high-dimensional spaces through optimized landmarks and adjustable lower bounds.

Findings

Memory-based methods improved by up to 90% in graph-based search.

Quantization-based search improved by up to 200%.

Disk-based methods reduced I/O costs by up to 58% and improved efficiency by 102%.

Abstract

High-dimensional vector similarity search (HVSS) is critical for many data processing and AI applications. However, traditional HVSS methods often require extensive data access for distance calculations, leading to inefficiencies. Triangle-inequality-based lower bound pruning is a widely used technique to reduce the number of data access in low-dimensional spaces but becomes less effective in high-dimensional settings. This is attributed to the "distance concentration" phenomenon, where the lower bounds derived from the triangle inequality become too small to be useful. To address this, we propose TRIM, which enhances the effectiveness of traditional triangle-inequality-based pruning in high-dimensional vector similarity search using two key ways: (1) optimizing landmark vectors used to form the triangles, and (2) relaxing the lower bounds derived from the triangle inequality, with the relaxation degree adjustable according to user's needs. TRIM is a versatile operation that can be seamlessly integrated into both memory-based (e.g., HNSW, IVFPQ) and disk-based (e.g., DiskANN) HVSS methods, reducing distance calculations and disk access. Extensive experiments show that TRIM enhances memory-based methods, improving graph-based search by up to 90% and quantization-based search by up to 200%, while achieving a pruning ratio of up to 99%. It also reduces I/O costs by up to 58% and improves efficiency by 102% for disk-based methods, while preserving high query accuracy.