Online Continual Knowledge Learning for Language Models

TL;DR

This paper introduces Online Continual Knowledge Learning (OCKL), a new challenge for updating language models with evolving world knowledge in real-time, along with benchmarks and evaluations to measure progress.

Contribution

It formulates the OCKL problem, proposes a benchmark and metrics, and evaluates existing methods, highlighting their limitations in dynamic knowledge updating.

Findings

Existing methods are insufficient for OCKL challenges.

Key factors affect the trade-off between knowledge acquisition and retention.

Established robust baselines for future research in OCKL.

Abstract

Large Language Models (LLMs) serve as repositories of extensive world knowledge, enabling them to perform tasks such as question-answering and fact-checking. However, this knowledge can become obsolete as global contexts change. In this paper, we introduce a novel problem in the realm of continual learning: Online Continual Knowledge Learning (OCKL). This problem formulation aims to manage the dynamic nature of world knowledge in LMs under real-time constraints. We propose a new benchmark and evaluation metric designed to measure both the rate of new knowledge acquisition and the retention of previously learned knowledge. Our empirical evaluation, conducted using a variety of state-of-the-art methods, establishes robust base-lines for OCKL. Our results reveal that existing continual learning approaches are unfortunately insufficient for tackling the unique challenges posed by OCKL. We identify key factors that influence the trade-off between knowledge acquisition and retention, thereby advancing our understanding of how to train LMs in a continually evolving environment.