The impact of model size on catastrophic forgetting in Online Continual Learning

TL;DR

This paper explores how the size of neural networks influences catastrophic forgetting in online continual learning, revealing that larger models do not necessarily perform better and may struggle more with adapting to new tasks.

Contribution

It provides empirical evidence that larger models do not always improve continual learning performance, challenging assumptions about model size and forgetting.

Findings

Larger models often struggle more with online continual learning.

Model size does not guarantee reduced catastrophic forgetting.

The relationship between model size and learning efficacy is nuanced.

Abstract

This study investigates the impact of model size on Online Continual Learning performance, with a focus on catastrophic forgetting. Employing ResNet architectures of varying sizes, the research examines how network depth and width affect model performance in class-incremental learning using the SplitCIFAR-10 dataset. Key findings reveal that larger models do not guarantee better Continual Learning performance; in fact, they often struggle more in adapting to new tasks, particularly in online settings. These results challenge the notion that larger models inherently mitigate catastrophic forgetting, highlighting the nuanced relationship between model size and Continual Learning efficacy. This study contributes to a deeper understanding of model scalability and its practical implications in Continual Learning scenarios.