LightCL: Compact Continual Learning with Low Memory Footprint For Edge Device

TL;DR

LightCL is a compact continual learning algorithm designed for edge devices that reduces memory usage by evaluating and compressing neural network components, maintaining generalizability while stabilizing features across tasks.

Contribution

The paper introduces LightCL, a novel continual learning method that evaluates and compresses neural network redundancy, maintaining generalizability with minimal resource consumption.

Findings

LightCL reduces memory footprint by up to 6.16 times.

It outperforms state-of-the-art continual learning methods.

Effective on edge devices with limited resources.

Abstract

Continual learning (CL) is a technique that enables neural networks to constantly adapt to their dynamic surroundings. Despite being overlooked for a long time, this technology can considerably address the customized needs of users in edge devices. Actually, most CL methods require huge resource consumption by the training behavior to acquire generalizability among all tasks for delaying forgetting regardless of edge scenarios. Therefore, this paper proposes a compact algorithm called LightCL, which evaluates and compresses the redundancy of already generalized components in structures of the neural network. Specifically, we consider two factors of generalizability, learning plasticity and memory stability, and design metrics of both to quantitatively assess generalizability of neural networks during CL. This evaluation shows that generalizability of different layers in a neural network exhibits a significant variation. Thus, we $\textit{Maintain Generalizability}$ by freezing generalized parts without the resource-intensive training process and $\textit{Memorize Feature Patterns}$ by stabilizing feature extracting of previous tasks to enhance generalizability for less-generalized parts with a little extra memory, which is far less than the reduction by freezing. Experiments illustrate that LightCL outperforms other state-of-the-art methods and reduces at most $\textbf{6.16$\times$}$ memory footprint. We also verify the effectiveness of LightCL on the edge device.