Smaller Is Better: An Analysis of Instance Quantity/Quality Trade-off in Rehearsal-based Continual Learning

TL;DR

This paper analyzes how the quantity and quality of stored instances affect rehearsal-based continual learning, finding that increasing instance quantity with compression often outperforms existing methods, especially under memory constraints.

Contribution

It introduces a comprehensive analysis of memory trade-offs in rehearsal-based continual learning, highlighting the effectiveness of heavily compressed instances and the role of different compression techniques.

Findings

Heavily compressed instances with high quantity outperform less compressed ones.

Deep encoders combined with extreme resizing excel in high-memory scenarios.

Extreme Learning Machines are advantageous in memory-constrained settings.

Abstract

The design of machines and algorithms capable of learning in a dynamically changing environment has become an increasingly topical problem with the increase of the size and heterogeneity of data available to learning systems. As a consequence, the key issue of Continual Learning has become that of addressing the stability-plasticity dilemma of connectionist systems, as they need to adapt their model without forgetting previously acquired knowledge. Within this context, rehearsal-based methods i.e., solutions in where the learner exploits memory to revisit past data, has proven to be very effective, leading to performance at the state-of-the-art. In our study, we propose an analysis of the memory quantity/quality trade-off adopting various data reduction approaches to increase the number of instances storable in memory. In particular, we investigate complex instance compression techniques such as deep encoders, but also trivial approaches such as image resizing and linear dimensionality reduction. Our findings suggest that the optimal trade-off is severely skewed toward instance quantity, where rehearsal approaches with several heavily compressed instances easily outperform state-of-the-art approaches with the same amount of memory at their disposal. Further, in high memory configurations, deep approaches extracting spatial structure combined with extreme resizing (of the order of $8\times8$ images) yield the best results, while in memory-constrained configurations where deep approaches cannot be used due to their memory requirement in training, Extreme Learning Machines (ELM) offer a clear advantage.