Architecture Matters in Continual Learning

TL;DR

This paper investigates how different neural network architectures influence continual learning performance, revealing that architectural choices significantly affect the balance between remembering past tasks and learning new ones.

Contribution

It demonstrates the importance of architecture selection in continual learning and provides practical guidelines for architectural decisions to enhance performance.

Findings

Architectural choices impact continual learning trade-offs.

Different architectures lead to varying forgetting and learning capabilities.

Recommendations for architecture design improve continual learning outcomes.

Abstract

A large body of research in continual learning is devoted to overcoming the catastrophic forgetting of neural networks by designing new algorithms that are robust to the distribution shifts. However, the majority of these works are strictly focused on the "algorithmic" part of continual learning for a "fixed neural network architecture", and the implications of using different architectures are mostly neglected. Even the few existing continual learning methods that modify the model assume a fixed architecture and aim to develop an algorithm that efficiently uses the model throughout the learning experience. However, in this work, we show that the choice of architecture can significantly impact the continual learning performance, and different architectures lead to different trade-offs between the ability to remember previous tasks and learning new ones. Moreover, we study the impact of various architectural decisions, and our findings entail best practices and recommendations that can improve the continual learning performance.