Advancing continual lifelong learning in neural information retrieval: definition, dataset, framework, and empirical evaluation

TL;DR

This paper introduces a systematic formulation, a new dataset, and a comprehensive framework for continual neural information retrieval, demonstrating how different models and strategies perform in lifelong learning scenarios.

Contribution

It provides the first well-defined task formulation, a multi-topic dataset, and an empirical evaluation framework for continual neural information retrieval.

Findings

Embedding-based models decline with topic shift and data volume increases.

Pretraining-based models are unaffected by topic shift and data volume.

Proper learning strategies mitigate catastrophic forgetting.

Abstract

Continual learning refers to the capability of a machine learning model to learn and adapt to new information, without compromising its performance on previously learned tasks. Although several studies have investigated continual learning methods for information retrieval tasks, a well-defined task formulation is still lacking, and it is unclear how typical learning strategies perform in this context. To address this challenge, a systematic task formulation of continual neural information retrieval is presented, along with a multiple-topic dataset that simulates continuous information retrieval. A comprehensive continual neural information retrieval framework consisting of typical retrieval models and continual learning strategies is then proposed. Empirical evaluations illustrate that the proposed framework can successfully prevent catastrophic forgetting in neural information retrieval and enhance performance on previously learned tasks. The results indicate that embedding-based retrieval models experience a decline in their continual learning performance as the topic shift distance and dataset volume of new tasks increase. In contrast, pretraining-based models do not show any such correlation. Adopting suitable learning strategies can mitigate the effects of topic shift and data augmentation.