CRISP: Correlation-Resilient Indexing via Subspace Partitioning

TL;DR

CRISP is a new high-dimensional ANN indexing framework that adaptively reduces preprocessing complexity and offers flexible query modes, achieving state-of-the-art throughput and efficiency on very high-dimensional datasets.

Contribution

CRISP introduces an adaptive, correlation-aware subspace partitioning method combined with a dual-mode query engine for efficient high-dimensional ANN search.

Findings

Achieves state-of-the-art query throughput on datasets up to 4096 dimensions.

Reduces preprocessing complexity compared to existing methods.

Demonstrates low memory usage and high efficiency in extensive evaluations.

Abstract

As the dimensionality of modern learned representations increases to thousands of dimensions, the state-of-the-art Approximate Nearest Neighbor (ANN) indices exhibit severe limitations. Graph-based methods (e.g., HNSW) suffer from prohibitive memory consumption and routing degradation, while recent randomized quantization and learned rotation approaches (e.g., RaBitQ, OPQ) impose significant preprocessing overheads. We introduce CRISP, a novel framework designed for ANN search in very-high-dimensional spaces. Unlike rigid pipelines that apply expensive orthogonal rotations indiscriminately, CRISP employs a lightweight, correlation- aware adaptive strategy that redistributes variance only when necessary, effectively reducing the preprocessing complexity. We couple this adaptive mechanism with a cache-coherent Compressed Sparse Row (CSR) index structure. Furthermore, CRISP incorporates a multi-stage dual-mode query engine: a Guaranteed Mode that preserves rigorous theoretical lower bounds on recall, and an Optimized Mode that leverages rank-based weighted scoring and early termination to reduce query latency. Extensive evaluation on datasets of very high dimensionality (up to 4096) demonstrates that CRISP achieves state-of-the-art query throughput, low construction costs, and peak memory efficiency.