Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approach

TL;DR

This paper introduces a memory-efficient, BP-free training method for neural networks on microcontrollers, using quantized zeroth-order gradient estimation and dimension reduction to enable effective training on resource-constrained edge devices.

Contribution

It proposes a novel BP-free training scheme with quantized zeroth-order methods and dimension reduction techniques suitable for microcontrollers, simplifying hardware design and reducing resource requirements.

Findings

Achieves comparable performance to traditional BP-based training on corrupted image data.

Enables dense full-model training on MCUs with 1024-KB SRAM.

Supports sparse training on MCUs with 256-KB SRAM.

Abstract

Back propagation (BP) is the default solution for gradient computation in neural network training. However, implementing BP-based training on various edge devices such as FPGA, microcontrollers (MCUs), and analog computing platforms face multiple major challenges, such as the lack of hardware resources, long time-to-market, and dramatic errors in a low-precision setting. This paper presents a simple BP-free training scheme on an MCU, which makes edge training hardware design as easy as inference hardware design. We adopt a quantized zeroth-order method to estimate the gradients of quantized model parameters, which can overcome the error of a straight-through estimator in a low-precision BP scheme. We further employ a few dimension reduction methods (e.g., node perturbation, sparse training) to improve the convergence of zeroth-order training. Experiment results show that our BP-free training achieves comparable performance as BP-based training on adapting a pre-trained image classifier to various corrupted data on resource-constrained edge devices (e.g., an MCU with 1024-KB SRAM for dense full-model training, or an MCU with 256-KB SRAM for sparse training). This method is most suitable for application scenarios where memory cost and time-to-market are the major concerns, but longer latency can be tolerated.