Do Pre-trained Models Benefit Equally in Continual Learning?

TL;DR

This paper explores how pre-trained models impact continual learning, revealing that pre-training can significantly alter algorithm performance and suggesting a new baseline for future research.

Contribution

It systematically investigates the role of pre-trained models in continual learning across different algorithms and scenarios, highlighting their complex effects and proposing a strong new baseline.

Findings

Pre-trained models can make underperforming CL algorithms competitive.

Less regularized CL algorithms benefit more from pre-training.

Stronger pre-trained models like CLIP do not always lead to better improvements.

Abstract

Existing work on continual learning (CL) is primarily devoted to developing algorithms for models trained from scratch. Despite their encouraging performance on contrived benchmarks, these algorithms show dramatic performance drops in real-world scenarios. Therefore, this paper advocates the systematic introduction of pre-training to CL, which is a general recipe for transferring knowledge to downstream tasks but is substantially missing in the CL community. Our investigation reveals the multifaceted complexity of exploiting pre-trained models for CL, along three different axes, pre-trained models, CL algorithms, and CL scenarios. Perhaps most intriguingly, improvements in CL algorithms from pre-training are very inconsistent an underperforming algorithm could become competitive and even state-of-the-art when all algorithms start from a pre-trained model. This indicates that the current paradigm, where all CL methods are compared in from-scratch training, is not well reflective of the true CL objective and desired progress. In addition, we make several other important observations, including that CL algorithms that exert less regularization benefit more from a pre-trained model; and that a stronger pre-trained model such as CLIP does not guarantee a better improvement. Based on these findings, we introduce a simple yet effective baseline that employs minimum regularization and leverages the more beneficial pre-trained model, coupled with a two-stage training pipeline. We recommend including this strong baseline in the future development of CL algorithms, due to its demonstrated state-of-the-art performance.