Continual Learning on a Data Diet

TL;DR

This paper investigates how selecting important samples, inspired by human cognition, can improve continual learning by enhancing accuracy, retention, and representations, through an empirical study of coreset selection techniques.

Contribution

It introduces an empirical evaluation of coreset selection in continual learning, revealing benefits in accuracy, retention, and representation quality.

Findings

Selective learning improves incremental accuracy

It enhances knowledge retention of previous tasks

Refines learned representations in continual learning

Abstract

Continual Learning (CL) methods usually learn from all available data. However, this is not the case in human cognition which efficiently focuses on key experiences while disregarding the redundant information. Similarly, not all data points in a dataset have equal potential; some can be more informative than others. This disparity may significantly impact the performance, as both the quality and quantity of samples directly influence the model's generalizability and efficiency. Drawing inspiration from this, we explore the potential of learning from important samples and present an empirical study for evaluating coreset selection techniques in the context of CL to stimulate research in this unexplored area. We train different continual learners on increasing amounts of selected samples and investigate the learning-forgetting dynamics by shedding light on the underlying mechanisms driving their improved stability-plasticity balance. We present several significant observations: learning from selectively chosen samples (i) enhances incremental accuracy, (ii) improves knowledge retention of previous tasks, and (iii) refines learned representations. This analysis contributes to a deeper understanding of selective learning strategies in CL scenarios.