A Simple Baseline for Stable and Plastic Neural Networks

TL;DR

This paper introduces RDBP, a simple and efficient baseline for continual learning in computer vision that balances plasticity and stability by combining activation modification and gradient scheduling, outperforming existing methods.

Contribution

The paper proposes RDBP, a novel, low-overhead method that unites ReLUDown and Decreasing Backpropagation to improve continual learning performance.

Findings

RDBP matches or exceeds state-of-the-art performance on Continual ImageNet.

RDBP reduces computational cost compared to existing methods.

RDBP provides a practical benchmark for future continual learning research.

Abstract

Continual learning in computer vision requires that models adapt to a continuous stream of tasks without forgetting prior knowledge, yet existing approaches often tip the balance heavily toward either plasticity or stability. We introduce RDBP, a simple, low-overhead baseline that unites two complementary mechanisms: ReLUDown, a lightweight activation modification that preserves feature sensitivity while preventing neuron dormancy, and Decreasing Backpropagation, a biologically inspired gradient-scheduling scheme that progressively shields early layers from catastrophic updates. Evaluated on the Continual ImageNet benchmark, RDBP matches or exceeds the plasticity and stability of state-of-the-art methods while reducing computational cost. RDBP thus provides both a practical solution for real-world continual learning and a clear benchmark against which future continual learning strategies can be measured.