Irrational Complex Rotations Empower Low-bit Optimizers

TL;DR

This paper introduces $ ext{pi}$-Quant, a novel optimizer state compression method using irrational number properties for memory-efficient training, achieving significant parameter reduction without accuracy loss.

Contribution

The paper presents a new parameter compression algorithm based on complex rotations and irrational numbers, enabling low-bit optimization with linear complexity.

Findings

Reduces parameter bit-width to 3.32 bits

Achieves 75% parameter scale reduction

Decreases GPU memory usage by 40%

Abstract

In this paper, we propose a novel optimizer state compression algorithm, namely $\pi$-Quant, which leverages the properties of irrational numbers (e.g., $\pi$) for memory-efficient training. The core idea is based on our mathematical findings, which show that a pair of parameters can be represented by a single rotation angle using the complex rotation scheme. Building on this insight, we map the parameters into a complex space and perform quantization using the corresponding rotation angles. To efficiently integrate it into optimization process, we develop an efficient system of geometric equations that computes the precise rotation angles with linear complexity. We evaluate $\pi$-Quant on a wide range of tasks. Our experiments show that it can reduce the bit-width of parameters to 3.32-bit, achieving a 75% reduction in parameter scale and a 40% decrease in GPU memory usage, all while maintaining full accuracy.