UnIte: Uncertainty-based Iterative Document Sampling for Domain Adaptation in Information Retrieval

TL;DR

UnIte introduces an uncertainty-based iterative sampling method for domain adaptation in neural information retrieval, improving pseudo query generation by focusing on model uncertainty.

Contribution

It proposes a novel sampling approach that filters documents by aleatoric uncertainty and prioritizes epistemic uncertainty, enhancing domain adaptation performance.

Findings

Achieved +2.45 and +3.49 nDCG@10 improvements on BEIR datasets.

Reduced training sample size to 4k while maintaining performance.

Demonstrated effectiveness on both small and large models.

Abstract

Unsupervised domain adaptation generalizes neural retrievers to an unseen domain by generating pseudo queries on target domain documents. The quality and efficiency of this adaptation critically depend on which documents are selected for pseudo query generation. The existing document sampling method focuses on diversity but fails to capture model uncertainty. In contrast, we propose **Un**certainty-based **Ite**rative Document Sampling (UnIte) addressing these limitations by (1) filtering documents with high aleatoric uncertainty and (2) prioritizing those with high epistemic uncertainty, maximizing the learning utility of the current model. We conducted extensive experiments on a large corpus of BEIR with small and large models, showing significant gains of +2.45 and +3.49 nDCG@10 with a smaller training sample size, 4k on average.