Beyond Bag-of-Patches: Learning Global Layout via Textual Supervision for Late-Interaction Visual Document Retrieval

TL;DR

This paper introduces a method to incorporate global layout information into visual document retrieval models using textual supervision, improving accuracy without altering inference procedures.

Contribution

It proposes a multimodal encoder that learns global layout embeddings from textual descriptions, enhancing late-interaction VDR models.

Findings

Model outperforms baseline by +2.4 nDCG@5 and +2.3 MAP@5 across datasets.

Global layout embeddings improve document relevance estimation.

Statistically significant improvements over comparable architectures.

Abstract

Visual Document Retrieval (VDR) models mostly rely on late interaction architectures, in which documents are represented by a set of local patch embeddings and then matched against query tokens. While efficient, this architecture prioritizes local similarity over global layout structure of documents to estimate relevancy between documents and query. In practice, this leads to errors as relevance originates from layout structure of documents with heterogeneous layouts combining figures, tables, and text. We make document layout learnable without changing inference. We propose a multimodal encoder that augments local patch representations with a global layout embedding, trained via textual descriptions encoding document layout information. Across four ViDoRe-v2 datasets, our model improves over the strongest architecturally comparable ColPali/ColQwen baseline by +2.4 nDCG@5 and +2.3 MAP@5, with statistically significant per-dataset gains over ColQwen.