Continual Deep Learning on the Edge via Stochastic Local Competition among Subnetworks

TL;DR

This paper presents a stochastic competition-based method for continual deep learning on edge devices, promoting sparsity in networks to reduce memory and computation while enabling efficient task-specific representations.

Contribution

It introduces a novel stochastic local competition mechanism that sparsifies weights and gradients, optimizing deep networks for resource-constrained edge environments.

Findings

Achieves significant sparsity in weights and gradients.

Reduces memory footprint and computational demand.

Enables efficient continual learning on edge devices.

Abstract

Continual learning on edge devices poses unique challenges due to stringent resource constraints. This paper introduces a novel method that leverages stochastic competition principles to promote sparsity, significantly reducing deep network memory footprint and computational demand. Specifically, we propose deep networks that comprise blocks of units that compete locally to win the representation of each arising new task; competition takes place in a stochastic manner. This type of network organization results in sparse task-specific representations from each network layer; the sparsity pattern is obtained during training and is different among tasks. Crucially, our method sparsifies both the weights and the weight gradients, thus facilitating training on edge devices. This is performed on the grounds of winning probability for each unit in a block. During inference, the network retains only the winning unit and zeroes-out all weights pertaining to non-winning units for the task at hand. Thus, our approach is specifically tailored for deployment on edge devices, providing an efficient and scalable solution for continual learning in resource-limited environments.