Momentum Posterior Regularization for Multi-hop Dense Retrieval

TL;DR

This paper introduces MoPo, a novel method for multi-hop dense retrieval that distills posterior knowledge into prior retrieval using momentum updates, significantly improving multi-hop QA performance.

Contribution

MoPo proposes a new posterior regularization approach with momentum updates to enhance knowledge distillation in multi-hop dense retrieval.

Findings

MoPo outperforms existing baselines on HotpotQA and StrategyQA.

Effective knowledge distillation improves multi-hop QA accuracy.

Momentum-based training stabilizes the distillation process.

Abstract

Multi-hop question answering (QA) often requires sequential retrieval (multi-hop retrieval), where each hop retrieves missing knowledge based on information from previous hops. To facilitate more effective retrieval, we aim to distill knowledge from a posterior retrieval, which has access to posterior information like an answer, into a prior retrieval used during inference when such information is unavailable. Unfortunately, current methods for knowledge distillation in one-time retrieval are ineffective for multi-hop QA due to two issues: 1) Posterior information is often defined as the response (i.e. the answer), which may not clearly connect to the query without intermediate retrieval; and 2) The large knowledge gap between prior and posterior retrievals makes existing distillation methods unstable, even resulting in performance loss. As such, we propose MoPo (Momentum Posterior Regularization) with two key innovations: 1) Posterior information of one hop is defined as a query-focus summary from the golden knowledge of the previous and current hops; 2) We develop an effective training strategy where the posterior retrieval is updated along with the prior retrieval via momentum moving average method, allowing smoother and effective distillation. Experiments on HotpotQA and StrategyQA demonstrate that MoPo outperforms existing baselines in both retrieval and downstream QA tasks.