Improving Representational Continuity via Continued Pretraining

TL;DR

This paper investigates continual representation learning, revealing that a transfer learning method called LP-FT outperforms traditional continual learning techniques in practical scenarios, achieving strong results across multiple benchmarks.

Contribution

It demonstrates that LP-FT, a transfer learning approach, surpasses existing continual learning methods in real-world settings and standard benchmarks, simplifying the process.

Findings

LP-FT outperforms naive training and other continual learning methods.

Strong continual learning baselines perform worse than naive training with standard adaptation.

LP-FT achieves state-of-the-art results on an NLP continual learning benchmark.

Abstract

We consider the continual representation learning setting: sequentially pretrain a model $M'$ on tasks $T_1, \ldots, T_T$, and then adapt $M'$ on a small amount of data from each task $T_i$ to check if it has forgotten information from old tasks. Under a kNN adaptation protocol, prior work shows that continual learning methods improve forgetting over naive training (SGD). In reality, practitioners do not use kNN classifiers -- they use the adaptation method that works best (e.g., fine-tuning) -- here, we find that strong continual learning baselines do worse than naive training. Interestingly, we find that a method from the transfer learning community (LP-FT) outperforms naive training and the other continual learning methods. Even with standard kNN evaluation protocols, LP-FT performs comparably with strong continual learning methods (while being simpler and requiring less memory) on three standard benchmarks: sequential CIFAR-10, CIFAR-100, and TinyImageNet. LP-FT also reduces forgetting in a real world satellite remote sensing dataset (FMoW), and a variant of LP-FT gets state-of-the-art accuracies on an NLP continual learning benchmark.