Continual Learning for Generative Retrieval over Dynamic Corpora

TL;DR

This paper introduces CLEVER, a continual learning model for generative retrieval over dynamic corpora, enabling incremental document indexing and retrieval without forgetting previous knowledge.

Contribution

The paper proposes a novel continual learning framework for generative retrieval, including Incremental Product Quantization and a memory-augmented mechanism for efficient and effective incremental document encoding.

Findings

CLEVER achieves high retrieval accuracy on dynamic datasets.

The model efficiently updates document representations with low computational cost.

Empirical results show CLEVER outperforms baseline methods in continual retrieval tasks.

Abstract

Generative retrieval (GR) directly predicts the identifiers of relevant documents (i.e., docids) based on a parametric model. It has achieved solid performance on many ad-hoc retrieval tasks. So far, these tasks have assumed a static document collection. In many practical scenarios, however, document collections are dynamic, where new documents are continuously added to the corpus. The ability to incrementally index new documents while preserving the ability to answer queries with both previously and newly indexed relevant documents is vital to applying GR models. In this paper, we address this practical continual learning problem for GR. We put forward a novel Continual-LEarner for generatiVE Retrieval (CLEVER) model and make two major contributions to continual learning for GR: (i) To encode new documents into docids with low computational cost, we present Incremental Product Quantization, which updates a partial quantization codebook according to two adaptive thresholds; and (ii) To memorize new documents for querying without forgetting previous knowledge, we propose a memory-augmented learning mechanism, to form meaningful connections between old and new documents. Empirical results demonstrate the effectiveness and efficiency of the proposed model.