Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines

TL;DR

This paper systematically categorizes continual learning scenarios, evaluates them with strong baselines, and highlights that simple methods can perform comparably to complex ones, suggesting the need for more challenging benchmarks.

Contribution

It provides a unified framework for evaluating continual learning scenarios and demonstrates that simple baselines are often competitive with advanced methods.

Findings

Simple baselines perform similarly to state-of-the-art methods

Certain scenarios are easier than previously thought

Recommendations for designing more challenging evaluation benchmarks

Abstract

Continual learning has received a great deal of attention recently with several approaches being proposed. However, evaluations involve a diverse set of scenarios making meaningful comparison difficult. This work provides a systematic categorization of the scenarios and evaluates them within a consistent framework including strong baselines and state-of-the-art methods. The results provide an understanding of the relative difficulty of the scenarios and that simple baselines (Adagrad, L2 regularization, and naive rehearsal strategies) can surprisingly achieve similar performance to current mainstream methods. We conclude with several suggestions for creating harder evaluation scenarios and future research directions. The code is available at https://github.com/GT-RIPL/Continual-Learning-Benchmark