Spatially-Grounded Document Retrieval via Patch-to-Region Relevance Propagation

TL;DR

This paper introduces a hybrid method combining patch-level visual similarity with OCR-extracted regions for precise, spatially-grounded document retrieval, improving localization accuracy and reducing context size without extra training.

Contribution

It formalizes coordinate mapping and relevance propagation between vision transformer patches and OCR regions, enabling spatial relevance filtering in document retrieval.

Findings

Achieves 59.7% hit rate at IoU@0.5 on BBox-DocVQA

Reduces context tokens by up to 52.3%

Operates without additional training

Abstract

Late-interaction multimodal retrieval models like ColPali achieve state-of-the-art document retrieval by embedding pages as images and computing fine-grained similarity between query tokens and visual patches. However, they operate at page-level granularity, limiting utility for retrieval-augmented generation (RAG) where precise context is paramount. Conversely, OCR-based systems extract structured text with bounding box coordinates but lack semantic grounding for relevance assessment. We propose a hybrid architecture that unifies these paradigms: using ColPali's patch-level similarity scores as spatial relevance filters over OCR-extracted regions. We formalize the coordinate mapping between vision transformer patch grids and OCR bounding boxes, introduce intersection metrics for relevance propagation, and establish theoretical bounds on area efficiency. We evaluate on BBox-DocVQA with ground-truth bounding boxes. For within-page localization (given correct page retrieval), ColQwen3-4B with percentile-50 thresholding achieves 59.7% hit rate at IoU@0.5 (84.4% at IoU@0.25, 35.8% at IoU@0.7), with mean IoU of 0.569, compared to ~6.7% for random region selection. Our approach reduces context tokens by 28.8% compared to returning all OCR regions and by 52.3% compared to full-page image tokens. Our approach operates at inference time without additional training. We release Snappy, an open-source implementation at https://github.com/athrael-soju/Snappy.