Ada-QPacknet -- adaptive pruning with bit width reduction as an efficient continual learning method without forgetting

TL;DR

Ada-QPacknet introduces an adaptive architecture-based continual learning method that combines pruning and bit-width reduction through quantisation, achieving efficient task-specific sub-networks with minimal forgetting.

Contribution

It is the first CL approach to integrate both pruning and quantisation for creating task-specific sub-networks, enhancing efficiency and performance.

Findings

Achieves similar accuracy with low-bit quantisation as floating-point models.

Outperforms popular CL strategies in task and class incremental scenarios.

Reduces model size significantly while maintaining performance.

Abstract

Continual Learning (CL) is a process in which there is still huge gap between human and deep learning model efficiency. Recently, many CL algorithms were designed. Most of them have many problems with learning in dynamic and complex environments. In this work new architecture based approach Ada-QPacknet is described. It incorporates the pruning for extracting the sub-network for each task. The crucial aspect in architecture based CL methods is theirs capacity. In presented method the size of the model is reduced by efficient linear and nonlinear quantisation approach. The method reduces the bit-width of the weights format. The presented results shows that low bit quantisation achieves similar accuracy as floating-point sub-network on a well-know CL scenarios. To our knowledge it is the first CL strategy which incorporates both compression techniques pruning and quantisation for generating task sub-networks. The presented algorithm was tested on well-known episode combinations and compared with most popular algorithms. Results show that proposed approach outperforms most of the CL strategies in task and class incremental scenarios.