Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework

TL;DR

This paper introduces a Prune-then-Merge framework for visual document retrieval that improves efficiency and compression without sacrificing semantic fidelity, outperforming existing methods across multiple datasets.

Contribution

The paper proposes a novel two-stage Prune-then-Merge framework that enhances multi-vector visual document retrieval efficiency and compression while maintaining high semantic accuracy.

Findings

Outperforms existing methods on 29 datasets

Extends near-lossless compression range

Maintains robust performance at high compression ratios

Abstract

Visual Document Retrieval (VDR), which aims to retrieve relevant pages within vast corpora of visually-rich documents, is of significance in current multimodal retrieval applications. The state-of-the-art multi-vector paradigm excels in performance but suffers from prohibitive overhead, a problem that current efficiency methods like pruning and merging address imperfectly, creating a difficult trade-off between compression rate and feature fidelity. To overcome this dilemma, we introduce Prune-then-Merge, a novel two-stage framework that synergizes these complementary approaches. Our method first employs an adaptive pruning stage to filter out low-information patches, creating a refined, high-signal set of embeddings. Subsequently, a hierarchical merging stage compresses this pre-filtered set, effectively summarizing semantic content without the noise-induced feature dilution seen in single-stage methods. Extensive experiments on 29 VDR datasets demonstrate that our framework consistently outperforms existing methods, significantly extending the near-lossless compression range and providing robust performance at high compression ratios.