Low-Complexity Inference in Continual Learning via Compressed Knowledge Transfer

TL;DR

This paper introduces two model compression frameworks using pruning and knowledge distillation to enable low-complexity inference in continual learning, especially for class-incremental tasks without task labels.

Contribution

It proposes novel, efficient frameworks for compressing large pre-trained models in continual learning, improving inference speed while maintaining accuracy.

Findings

Both frameworks outperform strong baselines in accuracy-efficiency trade-offs.

Pruning strategies effectively reduce model size with minimal accuracy loss.

Knowledge distillation transfers relevant knowledge to smaller models, enhancing efficiency.

Abstract

Continual learning (CL) aims to train models that can learn a sequence of tasks without forgetting previously acquired knowledge. A core challenge in CL is balancing stability -- preserving performance on old tasks -- and plasticity -- adapting to new ones. Recently, large pre-trained models have been widely adopted in CL for their ability to support both, offering strong generalization for new tasks and resilience against forgetting. However, their high computational cost at inference time limits their practicality in real-world applications, especially those requiring low latency or energy efficiency. To address this issue, we explore model compression techniques, including pruning and knowledge distillation (KD), and propose two efficient frameworks tailored for class-incremental learning (CIL), a challenging CL setting where task identities are unavailable during inference. The pruning-based framework includes pre- and post-pruning strategies that apply compression at different training stages. The KD-based framework adopts a teacher-student architecture, where a large pre-trained teacher transfers downstream-relevant knowledge to a compact student. Extensive experiments on multiple CIL benchmarks demonstrate that the proposed frameworks achieve a better trade-off between accuracy and inference complexity, consistently outperforming strong baselines. We further analyze the trade-offs between the two frameworks in terms of accuracy and efficiency, offering insights into their use across different scenarios.