GEM: A Native Graph-based Index for Multi-Vector Retrieval

TL;DR

GEM introduces a native graph-based indexing method for multi-vector retrieval that preserves semantic richness and significantly improves search speed over existing approaches.

Contribution

The paper presents GEM, a novel multi-vector indexing framework that constructs a proximity graph directly over vector sets, enabling efficient and accurate retrieval.

Findings

Up to 16x speedup over state-of-the-art methods

Maintains or improves retrieval accuracy

Effective in in-domain, out-of-domain, and multi-modal benchmarks

Abstract

In multi-vector retrieval, both queries and data are represented as sets of high-dimensional vectors, enabling finer-grained semantic matching and improving retrieval quality over single-vector approaches. However, its practical adoption is held back by the lack of effective indexing algorithms. Existing work, attempting to reuse standard single-vector indexes, often fails to preserve multi-vector semantics or remains slow. In this work, we present GEM, a native indexing framework for multi-vector representations. The core idea is to construct a proximity graph directly over vector sets, preserving their fine-grained semantics while enabling efficient navigation. First, GEM designs a set-level clustering scheme. It associates each vector set with only its most informative clusters, effectively reducing redundancy without hurting semantic coverage. Then, it builds local proximity graphs within clusters and bridges them into a globally navigable structure. To handle the non-metric nature of multi-vector similarity, GEM decouples the graph construction metric from the final relevance score and injects semantic shortcuts to guide efficient navigation toward relevant regions. At query time, GEM launches beam search from multiple entry points and prunes paths early using cluster cues. To further enhance efficiency, a quantized distance estimation technique is used for both indexing and search. Across in-domain, out-of-domain, and multi-modal benchmarks, GEM achieves up to 16x speedup over state-of-the-art methods while matching or improving accuracy.