Recasting Continual Learning as Sequence Modeling

TL;DR

This paper proposes a novel approach to continual learning by framing it as a sequence modeling problem, enabling the use of advanced sequence models like Transformers to improve continual learning performance.

Contribution

It introduces a new formulation connecting continual learning with sequence modeling and demonstrates the effectiveness of Transformers within this framework.

Findings

Transformers can be effectively used for continual learning.

Sequence modeling offers a versatile approach for multiple benchmarks.

The method performs well on both classification and regression tasks.

Abstract

In this work, we aim to establish a strong connection between two significant bodies of machine learning research: continual learning and sequence modeling. That is, we propose to formulate continual learning as a sequence modeling problem, allowing advanced sequence models to be utilized for continual learning. Under this formulation, the continual learning process becomes the forward pass of a sequence model. By adopting the meta-continual learning (MCL) framework, we can train the sequence model at the meta-level, on multiple continual learning episodes. As a specific example of our new formulation, we demonstrate the application of Transformers and their efficient variants as MCL methods. Our experiments on seven benchmarks, covering both classification and regression, show that sequence models can be an attractive solution for general MCL.