Disentangled Modeling of Domain and Relevance for Adaptable Dense Retrieval

TL;DR

This paper introduces Disentangled Dense Retrieval (DDR), a novel framework that enhances domain adaptation in dense retrieval models by disentangling domain-invariant and domain-specific features, leading to improved cross-domain retrieval performance.

Contribution

The paper proposes DDR, which separates domain-invariant relevance modeling from domain-specific adaptation, enabling effective unsupervised training of domain modules and improved cross-domain retrieval.

Findings

DDR outperforms strong dense retrieval baselines across multiple domains.

DDR significantly surpasses traditional retrieval methods in various scenarios.

Disentangling features enhances domain adaptation and retrieval effectiveness.

Abstract

Recent advance in Dense Retrieval (DR) techniques has significantly improved the effectiveness of first-stage retrieval. Trained with large-scale supervised data, DR models can encode queries and documents into a low-dimensional dense space and conduct effective semantic matching. However, previous studies have shown that the effectiveness of DR models would drop by a large margin when the trained DR models are adopted in a target domain that is different from the domain of the labeled data. One of the possible reasons is that the DR model has never seen the target corpus and thus might be incapable of mitigating the difference between the training and target domains. In practice, unfortunately, training a DR model for each target domain to avoid domain shift is often a difficult task as it requires additional time, storage, and domain-specific data labeling, which are not always available. To address this problem, in this paper, we propose a novel DR framework named Disentangled Dense Retrieval (DDR) to support effective and flexible domain adaptation for DR models. DDR consists of a Relevance Estimation Module (REM) for modeling domain-invariant matching patterns and several Domain Adaption Modules (DAMs) for modeling domain-specific features of multiple target corpora. By making the REM and DAMs disentangled, DDR enables a flexible training paradigm in which REM is trained with supervision once and DAMs are trained with unsupervised data. Comprehensive experiments in different domains and languages show that DDR significantly improves ranking performance compared to strong DR baselines and substantially outperforms traditional retrieval methods in most scenarios.