DocReRank: Single-Page Hard Negative Query Generation for Training Multi-Modal RAG Rerankers

TL;DR

This paper introduces a novel method for training multimodal rerankers by generating hard negative queries per page using an LLM-VLM pipeline, leading to improved retrieval performance.

Contribution

It proposes a new approach to hard negative mining by generating negative queries per page, enhancing diversity and control in training data for rerankers.

Findings

Generated negatives are more diverse and harder.

Rerankers trained with generated negatives outperform existing models.

Significant improvements in retrieval performance achieved.

Abstract

Rerankers play a critical role in multimodal Retrieval-Augmented Generation (RAG) by refining ranking of an initial set of retrieved documents. Rerankers are typically trained using hard negative mining, whose goal is to select pages for each query which rank high, but are actually irrelevant. However, this selection process is typically passive and restricted to what the retriever can find in the available corpus, leading to several inherent limitations. These include: limited diversity, negative examples which are often not hard enough, low controllability, and frequent false negatives which harm training. Our paper proposes an alternative approach: Single-Page Hard Negative Query Generation, which goes the other way around. Instead of retrieving negative pages per query, we generate hard negative queries per page. Using an automated LLM-VLM pipeline, and given a page and its positive query, we create hard negatives by rephrasing the query to be as similar as possible in form and context, yet not answerable from the page. This paradigm enables fine-grained control over the generated queries, resulting in diverse, hard, and targeted negatives. It also supports efficient false negative verification. Our experiments show that rerankers trained with data generated using our approach outperform existing models and significantly improve retrieval performance.