Studying Catastrophic Forgetting in Neural Ranking Models

TL;DR

This paper investigates how neural ranking models forget previously learned domains when trained on new data, revealing the extent of catastrophic forgetting and proposing a lifelong learning strategy to mitigate it.

Contribution

It introduces a cross-domain regularizer for neural IR models that reduces catastrophic forgetting and analyzes domain characteristics affecting forgetting.

Findings

Neural IR models suffer from significant catastrophic forgetting.

A lifelong learning strategy with a cross-domain regularizer mitigates forgetting.

Domain characteristics influence the degree of forgetting.

Abstract

Several deep neural ranking models have been proposed in the recent IR literature. While their transferability to one target domain held by a dataset has been widely addressed using traditional domain adaptation strategies, the question of their cross-domain transferability is still under-studied. We study here in what extent neural ranking models catastrophically forget old knowledge acquired from previously observed domains after acquiring new knowledge, leading to performance decrease on those domains. Our experiments show that the effectiveness of neuralIR ranking models is achieved at the cost of catastrophic forgetting and that a lifelong learning strategy using a cross-domain regularizer success-fully mitigates the problem. Using an explanatory approach built on a regression model, we also show the effect of domain characteristics on the rise of catastrophic forgetting. We believe that the obtained results can be useful for both theoretical and practical future work in neural IR.