How Weight Resampling and Optimizers Shape the Dynamics of Continual Learning and Forgetting in Neural Networks

TL;DR

This paper investigates how weight resampling ('zapping') and optimizer choices influence learning and forgetting dynamics in neural networks during continual and transfer learning, revealing their roles in recovery speed and task interference.

Contribution

It provides a detailed analysis of how weight resampling and optimizers affect learning and forgetting patterns in neural networks under continual learning scenarios.

Findings

Models with zapping recover faster after domain transfer.

Optimizer choice significantly impacts learning and forgetting dynamics.

Complex task interference patterns emerge during sequential learning.

Abstract

Recent work in continual learning has highlighted the beneficial effect of resampling weights in the last layer of a neural network (``zapping"). Although empirical results demonstrate the effectiveness of this approach, the underlying mechanisms that drive these improvements remain unclear. In this work, we investigate in detail the pattern of learning and forgetting that take place inside a convolutional neural network when trained in challenging settings such as continual learning and few-shot transfer learning, with handwritten characters and natural images. Our experiments show that models that have undergone zapping during training more quickly recover from the shock of transferring to a new domain. Furthermore, to better observe the effect of continual learning in a multi-task setting we measure how each individual task is affected. This shows that, not only zapping, but the choice of optimizer can also deeply affect the dynamics of learning and forgetting, causing complex patterns of synergy/interference between tasks to emerge when the model learns sequentially at transfer time.