Hadamard Domain Training with Integers for Class Incremental Quantized Learning

TL;DR

This paper introduces a Hadamard transform-based method for low-precision, integer-only training in continual learning, significantly reducing computational costs while maintaining high accuracy.

Contribution

It proposes a novel Hadamard transform technique enabling effective low-bit quantized training with integer matrix multiplications for continual learning.

Findings

Achieves less than 0.5% accuracy degradation on recognition tasks.

Quantizes all matrix inputs to 4 bits with 8-bit accumulators.

Demonstrates effectiveness on multiple datasets in class incremental learning.

Abstract

Continual learning is a desirable feature in many modern machine learning applications, which allows in-field adaptation and updating, ranging from accommodating distribution shift, to fine-tuning, and to learning new tasks. For applications with privacy and low latency requirements, the compute and memory demands imposed by continual learning can be cost-prohibitive for resource-constraint edge platforms. Reducing computational precision through fully quantized training (FQT) simultaneously reduces memory footprint and increases compute efficiency for both training and inference. However, aggressive quantization especially integer FQT typically degrades model accuracy to unacceptable levels. In this paper, we propose a technique that leverages inexpensive Hadamard transforms to enable low-precision training with only integer matrix multiplications. We further determine which tensors need stochastic rounding and propose tiled matrix multiplication to enable low-bit width accumulators. We demonstrate the effectiveness of our technique on several human activity recognition datasets and CIFAR100 in a class incremental learning setting. We achieve less than 0.5% and 3% accuracy degradation while we quantize all matrix multiplications inputs down to 4-bits with 8-bit accumulators.