Attribution-Based Neuron Utility for Plasticity Restoration in Deep Networks

TL;DR

This paper introduces GXD, a new utility measure based on gradient attribution, to improve neuron reset interventions for maintaining plasticity in deep networks during continual learning.

Contribution

It proposes GXD, a theoretically motivated utility measure that better aligns with functional cost, enhancing the reliability of neuron reset strategies in continual learning.

Findings

GXD outperforms existing utility measures in reset interventions.

Utility measures aligned with functional cost improve plasticity restoration.

GXD provides a practical approach to robust continual learning.

Abstract

Continual learning research attempts to conserve two fundamental capabilities: new knowledge acquisition and the preservation of previously acquired knowledge. While knowledge in this case can be measured through performance over an implicit or explicit task space, model plasticity generally concerns adaptability as data distributions evolve. Though much of the literature has focused on catastrophic forgetting, deep networks can also suffer from loss of plasticity, becoming progressively harder to update under continued training. Recent research has identified multiple mechanisms underlying this phenomenon, including neuron saturation, parameter norm growth, and loss of useful curvature directions. Adaptive reset-based interventions, which selectively reinitialize low-utility network parameters, have emerged as practical solutions to restore trainability. Existing utility measures used to guide resets, such as activation magnitude, contribution utility, or gradient-based activity, rely on proxy signals that can become misaligned with the intervention they are meant to guide. In this paper, we introduce gradient times difference from reference (GXD), a theoretically motivated utility measure based on reference-based gradient attribution that estimates the first-order functional cost of replacing a unit. Our results show that utility measures aligned with the functional cost of the reset can make interventions more reliable in settings where existing reset criteria degrade. GXD reframes adaptive resetting as an intervention cost estimation problem, providing a practical path toward more robust continual learning systems.