Pushing the Limits of Distillation-Based Continual Learning via Classifier-Proximal Lightweight Plugins

TL;DR

This paper introduces DLC, a plugin-based method for distillation-based continual learning that improves accuracy with minimal additional parameters by using lightweight residual plugins and importance weighting.

Contribution

The paper proposes a novel plugin extension paradigm called DLC that enhances distillation-based continual learning through classifier-proximal lightweight residual plugins.

Findings

DLC achieves an 8% accuracy improvement on large-scale benchmarks.

DLC adds only 4% more parameters to the backbone.

DLC is compatible with other continual learning enhancements.

Abstract

Continual learning requires models to learn continuously while preserving prior knowledge under evolving data streams. Distillation-based methods are appealing for retaining past knowledge in a shared single-model framework with low storage overhead. However, they remain constrained by the stability-plasticity dilemma: knowledge acquisition and preservation are still optimized through coupled objectives, and existing enhancement methods do not alter this underlying bottleneck. To address this issue, we propose a plugin extension paradigm termed Distillation-aware Lightweight Components (DLC) for distillation-based CL. DLC deploys lightweight residual plugins into the base feature extractor's classifier-proximal layer, enabling semantic-level residual correction for better classification accuracy while minimizing disruption to the overall feature extraction process. During inference, plugin-enhanced representations are aggregated to produce classification predictions. To mitigate interference from non-target plugins, we further introduce a lightweight weighting unit that learns to assign importance scores to different plugin-enhanced representations. DLC could deliver a significant 8% accuracy gain on large-scale benchmarks while introducing only a 4% increase in backbone parameters, highlighting its exceptional efficiency. Moreover, DLC is compatible with other plug-and-play CL enhancements and delivers additional gains when combined with them.