HOP to the Next Tasks and Domains for Continual Learning in NLP

TL;DR

This paper introduces HOP, a novel continual learning framework for NLP that enables models to transfer knowledge across tasks and domains by using adapters, high-order moments, and specialized auxiliary heads, improving performance across multiple benchmarks.

Contribution

HOP is a new general framework for continual learning in NLP that allows hopping across tasks and domains using adapters and high-order statistical analysis.

Findings

HOP outperforms existing methods on 4 NLP applications.

Effective in 5 benchmark datasets with 2 CL setups.

Demonstrates robust knowledge transfer across tasks and domains.

Abstract

Continual Learning (CL) aims to learn a sequence of problems (i.e., tasks and domains) by transferring knowledge acquired on previous problems, whilst avoiding forgetting of past ones. Different from previous approaches which focused on CL for one NLP task or domain in a specific use-case, in this paper, we address a more general CL setting to learn from a sequence of problems in a unique framework. Our method, HOP, permits to hop across tasks and domains by addressing the CL problem along three directions: (i) we employ a set of adapters to generalize a large pre-trained model to unseen problems, (ii) we compute high-order moments over the distribution of embedded representations to distinguish independent and correlated statistics across different tasks and domains, (iii) we process this enriched information with auxiliary heads specialized for each end problem. Extensive experimental campaign on 4 NLP applications, 5 benchmarks and 2 CL setups demonstrates the effectiveness of our HOP.