Semi-Supervised Lifelong Language Learning

TL;DR

This paper introduces semi-supervised lifelong language learning (SSLL), enabling models to learn sequential language tasks using both labeled and unlabeled data, addressing catastrophic forgetting and improving knowledge transfer.

Contribution

It proposes a novel SSLL setting and a model with task-specific modules, virtual supervision, and backward augmentation to leverage unlabeled data effectively.

Findings

Model outperforms baselines on various language tasks.

Unlabeled data enhances knowledge transfer and reduces forgetting.

The approach demonstrates the effectiveness of semi-supervised learning in lifelong language tasks.

Abstract

Lifelong learning aims to accumulate knowledge and alleviate catastrophic forgetting when learning tasks sequentially. However, existing lifelong language learning methods only focus on the supervised learning setting. Unlabeled data, which can be easily accessed in real-world scenarios, are underexplored. In this paper, we explore a novel setting, semi-supervised lifelong language learning (SSLL), where a model learns sequentially arriving language tasks with both labeled and unlabeled data. We propose an unlabeled data enhanced lifelong learner to explore SSLL. Specially, we dedicate task-specific modules to alleviate catastrophic forgetting and design two modules to exploit unlabeled data: (1) a virtual supervision enhanced task solver is constructed on a teacher-student framework to mine the underlying knowledge from unlabeled data; and (2) a backward augmented learner is built to encourage knowledge transfer from newly arrived unlabeled data to previous tasks. Experimental results on various language tasks demonstrate our model's effectiveness and superiority over competitive baselines under the new setting SSLL.