HLQ: Fast and Efficient Backpropagation via Hadamard Low-rank Quantization

TL;DR

HLQ introduces a novel low-rank quantization method for backpropagation that significantly reduces memory and computation costs in training deep neural networks without compromising accuracy.

Contribution

The paper proposes Hadamard Low-rank Quantization (HLQ), a new approach combining 4-bit activation gradient quantization and low-rank weight gradient approximation to optimize backpropagation efficiency.

Findings

Achieves significant memory savings during training.

Provides acceleration on real GPUs with negligible accuracy loss.

Effective for both training from scratch and fine-tuning.

Abstract

With the rapid increase in model size and the growing importance of various fine-tuning applications, lightweight training has become crucial. Since the backward pass is twice as expensive as the forward pass, optimizing backpropagation is particularly important. However, modifications to this process can lead to suboptimal convergence, so training optimization should minimize perturbations, which is a highly challenging task. In this study, we introduce a novel optimization strategy called Hadamard Low-rank Quantization (HLQ), focusing on reducing the cost of backpropagation in convolutional and linear layers. We first analyze the sensitivity of gradient computation with respect to activation and weight, and judiciously design the HLQ pipeline to apply 4-bit Hadamard quantization to the activation gradient and Hadamard low-rank approximation to the weight gradient. This combination was found to be the best for maximizing benefits, and our extensive experiments demonstrate the outstanding performance of HLQ in both training from scratch and fine-tuning, achieving significant memory savings and acceleration on real GPUs with negligible quality degradation.