Reproducibility, Replicability, and Insights into Visual Document Retrieval with Late Interaction

TL;DR

This paper evaluates the reproducibility and effectiveness of late interaction mechanisms in Visual Document Retrieval, confirming their performance benefits while analyzing computational costs and robustness across datasets.

Contribution

It systematically assesses the impact of late interaction in VDR models, providing insights into their performance, efficiency, and robustness across multiple models and datasets.

Findings

Late interaction improves retrieval effectiveness significantly.

Late interaction introduces computational inefficiencies during inference.

Query-patch matching is indirect and relies on visual similarity.

Abstract

Visual Document Retrieval (VDR) is an emerging research area that focuses on encoding and retrieving document images directly, bypassing the dependence on Optical Character Recognition (OCR) for document search. A recent advance in VDR was introduced by ColPali, which significantly improved retrieval effectiveness through a late interaction mechanism. ColPali's approach demonstrated substantial performance gains over existing baselines that do not use late interaction on an established benchmark. In this study, we investigate the reproducibility and replicability of VDR methods with and without late interaction mechanisms by systematically evaluating their performance across multiple pre-trained vision-language models. Our findings confirm that late interaction yields considerable improvements in retrieval effectiveness; however, it also introduces computational inefficiencies during inference. Additionally, we examine the adaptability of VDR models to textual inputs and assess their robustness across text-intensive datasets within the proposed benchmark, particularly when scaling the indexing mechanism. Furthermore, our research investigates the specific contributions of late interaction by looking into query-patch matching in the context of visual document retrieval. We find that although query tokens cannot explicitly match image patches as in the text retrieval scenario, they tend to match the patch contains visually similar tokens or their surrounding patches.