ColMate: Contrastive Late Interaction and Masked Text for Multimodal Document Retrieval

TL;DR

ColMate is a novel multimodal document retrieval model that employs OCR-based pretraining, masked contrastive learning, and late interaction scoring, achieving significant improvements over existing methods especially in out-of-domain scenarios.

Contribution

The paper introduces ColMate, a new multimodal retrieval model with specialized pretraining and scoring mechanisms tailored for multimodal documents, addressing limitations of text-only adapted methods.

Findings

Achieves 3.61% improvement on ViDoRe V2 benchmark.

Demonstrates stronger generalization to out-of-domain data.

Utilizes OCR-based pretraining and masked contrastive learning.

Abstract

Retrieval-augmented generation has proven practical when models require specialized knowledge or access to the latest data. However, existing methods for multimodal document retrieval often replicate techniques developed for text-only retrieval, whether in how they encode documents, define training objectives, or compute similarity scores. To address these limitations, we present ColMate, a document retrieval model that bridges the gap between multimodal representation learning and document retrieval. ColMate utilizes a novel OCR-based pretraining objective, a self-supervised masked contrastive learning objective, and a late interaction scoring mechanism more relevant to multimodal document structures and visual characteristics. ColMate obtains 3.61% improvements over existing retrieval models on the ViDoRe V2 benchmark, demonstrating stronger generalization to out-of-domain benchmarks.