Critical Patch-Aware Sparse Prompting with Decoupled Training for Continual Learning on the Edge

TL;DR

This paper introduces CPS-Prompt, a sparse prompting method for continual learning on edge devices that reduces memory and computation during training while maintaining high accuracy.

Contribution

It proposes a novel framework combining critical patch sampling and decoupled prompt training to improve training efficiency on resource-constrained edge devices.

Findings

CPS-Prompt reduces peak memory, training time, and energy consumption by about 1.6x compared to CODA-Prompt.

Maintains accuracy within 2% of the state-of-the-art C-Prompt on benchmarks.

Demonstrates effectiveness on three public benchmarks and real edge hardware.

Abstract

Continual learning (CL) on edge devices requires not only high accuracy but also training-time efficiency to support on-device adaptation under strict memory and computational constraints. While prompt-based continual learning (PCL) is parameter-efficient and achieves competitive accuracy, prior work has focused mainly on accuracy or inference-time performance, often overlooking the memory and computational costs of on-device training. In this paper, we propose CPS-Prompt, a critical patch-aware sparse prompting framework that explicitly targets training-time memory usage and computational cost by integrating critical patch sampling (CPS) for task-aware token reduction and decoupled prompt and classifier training (DPCT) to reduce backpropagation overhead. Experiments on three public benchmarks and real edge hardware show that CPS-Prompt improves peak memory, training time, and energy efficiency by about 1.6x over the balanced CODA-Prompt baseline, while maintaining accuracy within 2% of the state-of-the-art C-Prompt on average and remaining competitive with CODA-Prompt in accuracy. The code is available at https://github.com/laymond1/cps-prompt.