Continual Lifelong Learning in Natural Language Processing: A Survey

TL;DR

This survey reviews the challenges, methods, evaluation techniques, and future directions for continual lifelong learning in NLP, emphasizing the unique difficulties posed by natural language's ambiguity and complexity.

Contribution

It provides a comprehensive overview of current CL approaches in NLP, critically analyzes evaluation methods, and outlines future research directions.

Findings

Major challenges in CL for NLP identified

Current neural network methods reviewed

Evaluation datasets and methods critically analyzed

Abstract

Continual learning (CL) aims to enable information systems to learn from a continuous data stream across time. However, it is difficult for existing deep learning architectures to learn a new task without largely forgetting previously acquired knowledge. Furthermore, CL is particularly challenging for language learning, as natural language is ambiguous: it is discrete, compositional, and its meaning is context-dependent. In this work, we look at the problem of CL through the lens of various NLP tasks. Our survey discusses major challenges in CL and current methods applied in neural network models. We also provide a critical review of the existing CL evaluation methods and datasets in NLP. Finally, we present our outlook on future research directions.