Adaptive Variance-Penalized Continual Learning with Fisher Regularization

TL;DR

This paper introduces a novel continual learning method that uses Fisher-weighted variance regularization to dynamically balance stability and plasticity, significantly reducing catastrophic forgetting in neural networks.

Contribution

It proposes a Fisher regularization-based framework that adaptively penalizes parameter variance, improving continual learning performance over existing methods.

Findings

Outperforms Variational Continual Learning and Elastic Weight Consolidation on benchmarks.

Effectively maintains knowledge across tasks with reduced forgetting.

Enhances model accuracy and stability in sequential learning scenarios.

Abstract

The persistent challenge of catastrophic forgetting in neural networks has motivated extensive research in continual learning . This work presents a novel continual learning framework that integrates Fisher-weighted asymmetric regularization of parameter variances within a variational learning paradigm. Our method dynamically modulates regularization intensity according to parameter uncertainty, achieving enhanced stability and performance. Comprehensive evaluations on standard continual learning benchmarks including SplitMNIST, PermutedMNIST, and SplitFashionMNIST demonstrate substantial improvements over existing approaches such as Variational Continual Learning and Elastic Weight Consolidation . The asymmetric variance penalty mechanism proves particularly effective in maintaining knowledge across sequential tasks while improving model accuracy. Experimental results show our approach not only boosts immediate task performance but also significantly mitigates knowledge degradation over time, effectively addressing the fundamental challenge of catastrophic forgetting in neural networks