Low-redundancy Distillation for Continual Learning

TL;DR

LoRD introduces a novel continual learning method inspired by brain mechanisms, reducing redundancy in models and rehearsal samples to improve training efficiency and accuracy.

Contribution

It proposes Low-redundancy Distillation (LoRD), a new CL approach that compresses models and optimizes rehearsal strategies to enhance efficiency without sacrificing performance.

Findings

LoRD achieves higher accuracy than existing methods.

LoRD reduces training FLOPs significantly.

It effectively balances training efficiency and model performance.

Abstract

Continual learning (CL) aims to learn new tasks without erasing previous knowledge. However, current CL methods primarily emphasize improving accuracy while often neglecting training efficiency, which consequently restricts their practical application. Drawing inspiration from the brain's contextual gating mechanism, which selectively filters neural information and continuously updates past memories, we propose Low-redundancy Distillation (LoRD), a novel CL method that enhances model performance while maintaining training efficiency. This is achieved by eliminating redundancy in three aspects of CL: student model redundancy, teacher model redundancy, and rehearsal sample redundancy. By compressing the learnable parameters of the student model and pruning the teacher model, LoRD facilitates the retention and optimization of prior knowledge, effectively decoupling task-specific knowledge without manually assigning isolated parameters for each task. Furthermore, we optimize the selection of rehearsal samples and refine rehearsal frequency to improve training efficiency. Through a meticulous design of distillation and rehearsal strategies, LoRD effectively balances training efficiency and model precision. Extensive experimentation across various benchmark datasets and environments demonstrates LoRD's superiority, achieving the highest accuracy with the lowest training FLOPs.