Towards Continual Knowledge Learning of Language Models

TL;DR

This paper introduces a new continual learning framework for language models, focusing on maintaining and updating world knowledge over time, and provides a benchmark to evaluate such models' ability to retain and acquire knowledge.

Contribution

It formulates the Continual Knowledge Learning problem, creates a benchmark and metrics, and demonstrates the challenges and importance of maintaining evolving knowledge in language models.

Findings

CKL presents unique challenges not seen in previous CL setups.

Parameter expansion is crucial for retaining and learning knowledge simultaneously.

CKL highlights critical causes of knowledge forgetting in language models.

Abstract

Large Language Models (LMs) are known to encode world knowledge in their parameters as they pretrain on a vast amount of web corpus, which is often utilized for performing knowledge-dependent downstream tasks such as question answering, fact-checking, and open dialogue. In real-world scenarios, the world knowledge stored in the LMs can quickly become outdated as the world changes, but it is non-trivial to avoid catastrophic forgetting and reliably acquire new knowledge while preserving invariant knowledge. To push the community towards better maintenance of ever-changing LMs, we formulate a new continual learning (CL) problem called Continual Knowledge Learning (CKL). We construct a new benchmark and metric to quantify the retention of time-invariant world knowledge, the update of outdated knowledge, and the acquisition of new knowledge. We adopt applicable recent methods from literature to create several strong baselines. Through extensive experiments, we find that CKL exhibits unique challenges that are not addressed in previous CL setups, where parameter expansion is necessary to reliably retain and learn knowledge simultaneously. By highlighting the critical causes of knowledge forgetting, we show that CKL is a challenging and important problem that helps us better understand and train ever-changing LMs. The benchmark datasets, evaluation script, and baseline code to reproduce our results are available at https://github.com/joeljang/continual-knowledge-learning.