Multi-Vector Index Compression in Any Modality

TL;DR

This paper introduces novel index compression techniques for multi-vector retrieval across various modalities, significantly reducing storage costs while maintaining or improving retrieval performance.

Contribution

It proposes four index compression methods, including a new attention-guided clustering approach, applicable to text, images, and videos, with comprehensive evaluation across multiple datasets.

Findings

Attention-guided clustering outperforms other compression methods

Compressed indexes achieve comparable or better retrieval performance

Methods are effective across text, visual, and video modalities

Abstract

We study efficient multi-vector retrieval for late interaction in any modality. Late interaction has emerged as a dominant paradigm for information retrieval in text, images, visual documents, and videos, but its computation and storage costs grow linearly with document length, making it costly for image-, video-, and audio-rich corpora. To address this limitation, we explore query-agnostic methods for compressing multi-vector document representations under a constant vector budget. We introduce four approaches for index compression: sequence resizing, memory tokens, hierarchical pooling, and a novel attention-guided clustering (AGC). AGC uses an attention-guided mechanism to identify the most semantically salient regions of a document as cluster centroids and to weight token aggregation. Evaluating these methods on retrieval tasks spanning text (BEIR), visual-document (ViDoRe), and video (MSR-VTT, MultiVENT 2.0), we show that attention-guided clustering consistently outperforms other parameterized compression methods (sequence resizing and memory tokens), provides greater flexibility in index size than non-parametric hierarchical clustering, and achieves competitive or improved performance compared to a full, uncompressed index. The source code is available at: github.com/hanxiangqin/omni-col-press.