When Should Dense Retrievers Be Updated in Evolving Corpora? Detecting Out-of-Distribution Corpora Using GradNormIR

TL;DR

This paper presents GradNormIR, an unsupervised method to detect out-of-distribution corpora for dense retrievers, enabling timely updates to maintain retrieval performance in evolving document collections.

Contribution

It introduces a novel task of predicting corpus OOD status and proposes GradNormIR, a gradient norm-based approach for proactive retriever update detection.

Findings

GradNormIR effectively detects OOD corpora in experiments.

Timely updates using GradNormIR improve retrieval robustness.

Method outperforms existing approaches on BEIR benchmark.

Abstract

Dense retrievers encode texts into embeddings to efficiently retrieve relevant documents from large databases in response to user queries. However, real-world corpora continually evolve, leading to a shift from the original training distribution of the retriever. Without timely updates or retraining, indexing newly emerging documents can degrade retrieval performance for future queries. Thus, identifying when a dense retriever requires an update is critical for maintaining robust retrieval systems. In this paper, we propose a novel task of predicting whether a corpus is out-of-distribution (OOD) relative to a dense retriever before indexing. Addressing this task allows us to proactively manage retriever updates, preventing potential retrieval failures. We introduce GradNormIR, an unsupervised approach that leverages gradient norms to detect OOD corpora effectively. Experiments on the BEIR benchmark demonstrate that GradNormIR enables timely updates of dense retrievers in evolving document collections, significantly enhancing retrieval robustness and efficiency.