TinySubNets: An efficient and low capacity continual learning strategy

TL;DR

TinySubNets (TSN) introduces a novel architectural continual learning strategy that combines pruning, adaptive quantization, and weight sharing to efficiently utilize model capacity, improve accuracy, and reduce resource consumption across tasks.

Contribution

TSN presents a new approach integrating pruning, quantization, and weight sharing to enhance capacity utilization and performance in continual learning.

Findings

TSN outperforms existing methods in accuracy on benchmark datasets.

TSN significantly improves model capacity exploitation.

The strategy reduces computational resource requirements.

Abstract

Continual Learning (CL) is a highly relevant setting gaining traction in recent machine learning research. Among CL works, architectural and hybrid strategies are particularly effective due to their potential to adapt the model architecture as new tasks are presented. However, many existing solutions do not efficiently exploit model sparsity, and are prone to capacity saturation due to their inefficient use of available weights, which limits the number of learnable tasks. In this paper, we propose TinySubNets (TSN), a novel architectural CL strategy that addresses the issues through the unique combination of pruning with different sparsity levels, adaptive quantization, and weight sharing. Pruning identifies a subset of weights that preserve model performance, making less relevant weights available for future tasks. Adaptive quantization allows a single weight to be separated into multiple parts which can be assigned to different tasks. Weight sharing between tasks boosts the exploitation of capacity and task similarity, allowing for the identification of a better trade-off between model accuracy and capacity. These features allow TSN to efficiently leverage the available capacity, enhance knowledge transfer, and reduce computational resource consumption. Experimental results involving common benchmark CL datasets and scenarios show that our proposed strategy achieves better results in terms of accuracy than existing state-of-the-art CL strategies. Moreover, our strategy is shown to provide a significantly improved model capacity exploitation. Code released at: https://github.com/lifelonglab/tinysubnets.