LITTA: Late-Interaction and Test-Time Alignment for Visually-Grounded Multimodal Retrieval

TL;DR

LITTA enhances multimodal document retrieval by generating query variants with a large language model and aggregating results, significantly improving accuracy and robustness without retraining the retriever.

Contribution

LITTA introduces a test-time query expansion method using large language models and late-interaction scoring to improve multimodal retrieval performance.

Findings

Multi-query retrieval improves top-k accuracy, recall, and MRR.

LITTA achieves large gains in domains with high visual and semantic variability.

The accuracy-efficiency trade-off is controllable by the number of query variants.

Abstract

Retrieving relevant evidence from visually rich documents such as textbooks, technical reports, and manuals is challenging due to long context, complex layouts, and weak lexical overlap between user questions and supporting pages. We propose LITTA, a query-expansion-centric retrieval framework for evidence page retrieval that improves multimodal document retrieval without retriever retraining. Given a user query, LITTA generates complementary query variants using a large language model and retrieves candidate pages for each variant using a frozen vision retriever with late-interaction scoring. Candidates from expanded queries are then aggregated through reciprocal rank fusion to improve evidence coverage and reduce sensitivity to any single phrasing. This simple test-time strategy significantly improves retrieval robustness while remaining compatible with existing multimodal embedding indices. We evaluate LITTA on visually grounded document retrieval tasks across three domains: computer science, pharmaceuticals, and industrial manuals. Multi-query retrieval consistently improves top-k accuracy, recall, and MRR compared to single-query retrieval, with particularly large gains in domains with high visual and semantic variability. Moreover, the accuracy-efficiency trade-off is directly controllable by the number of query variants, making LITTA practical for deployment under latency constraints. These results demonstrate that query expansion provides a simple yet effective mechanism for improving visually grounded multimodal retrieval.