Towards Robust Evaluations of Continual Learning

TL;DR

This paper critiques current continual learning evaluation methods, highlighting their flaws, proposing new desiderata and experiment designs to improve assessment accuracy, and urging the community to refocus research efforts.

Contribution

It identifies shortcomings in existing evaluation practices, introduces criteria for better assessments, and proposes improved experimental designs for continual learning research.

Findings

Current evaluations can mislead about approach effectiveness.

Proposed new experiment designs better assess continual learning methods.

Call for community to adopt improved evaluation standards.

Abstract

Experiments used in current continual learning research do not faithfully assess fundamental challenges of learning continually. Instead of assessing performance on challenging and representative experiment designs, recent research has focused on increased dataset difficulty, while still using flawed experiment set-ups. We examine standard evaluations and show why these evaluations make some continual learning approaches look better than they are. We introduce desiderata for continual learning evaluations and explain why their absence creates misleading comparisons. Based on our desiderata we then propose new experiment designs which we demonstrate with various continual learning approaches and datasets. Our analysis calls for a reprioritization of research effort by the community.