FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability-Plasticity Tradeoff

TL;DR

FIRE is a novel reinitialization method for deep neural networks that explicitly balances stability and plasticity by optimizing weight proximity and isotropy, improving continual learning performance.

Contribution

The paper introduces FIRE, a principled reinitialization approach that balances stability and plasticity through a constrained optimization, outperforming existing methods across multiple domains.

Findings

FIRE outperforms naive training and standard reinitialization in continual learning tasks.

FIRE effectively balances stability and plasticity in diverse domains.

FIRE's optimization approach is efficiently approximated by Newton-Schulz iteration.

Abstract

Deep neural networks trained on nonstationary data must balance stability (i.e., retaining prior knowledge) and plasticity (i.e., adapting to new tasks). Standard reinitialization methods, which reinitialize weights toward their original values, are widely used but difficult to tune: conservative reinitializations fail to restore plasticity, while aggressive ones erase useful knowledge. We propose FIRE, a principled reinitialization method that explicitly balances the stability-plasticity tradeoff. FIRE quantifies stability through Squared Frobenius Error (SFE), measuring proximity to past weights, and plasticity through Deviation from Isometry (DfI), reflecting weight isotropy. The reinitialization point is obtained by solving a constrained optimization problem, minimizing SFE subject to DfI being zero, which is efficiently approximated by Newton-Schulz iteration. FIRE is evaluated on continual visual learning (CIFAR-10 with ResNet-18), language modeling (OpenWebText with GPT-0.1B), and reinforcement learning (HumanoidBench with SAC and Atari games with DQN). Across all domains, FIRE consistently outperforms both naive training without intervention and standard reinitialization methods, demonstrating effective balancing of the stability-plasticity tradeoff.