STQuant: Spatio-Temporal Adaptive Framework for Optimizer Quantization in Large Multimodal Model Training

TL;DR

STQuant is a novel distributed training framework that adaptively allocates precision for optimizer states across layers and steps, significantly reducing memory usage while maintaining model accuracy.

Contribution

It introduces a dynamic precision allocation method with a provably near-optimal factor selection and a linear complexity transition algorithm for large multimodal model training.

Findings

Reduces optimizer-state memory by 84.4% on GPT-2 and ViT.

Achieves an average bit-width of 5.1 bits with minimal accuracy loss.

Requires only O(N/K) computational overhead and O(1) extra space.

Abstract

Quantization is an effective way to reduce the memory cost of large-scale model training. However, most existing methods adopt fixed-precision policies, which ignore the fact that optimizer-state distributions vary significantly across layers and training steps. Such uniform designs often introduce noticeable accuracy degradation. To move beyond fixed quantization, we propose STQuant, a distributed training framework that reduces the memory footprint of optimizer states via dynamic precision allocation across layers, state variables, and training steps, while maintaining model quality. Naively applying dynamic quantization during training is challenging for two reasons. First, optimizer states are numerically sensitive, and quantization noise can destabilize quality. Second, jointly considering multiple states and layers induces a large combinatorial search space. STQuant addresses these challenges with two key techniques: 1) a provably near-optimal factor selection strategy that accurately identifies the most influential factors for precision adaptation. 2) a dynamic transition decision algorithm that reduces the search cost from exponential to linear complexity. Experiments on GPT-2 and ViT show that STQuant reduces optimizer-state memory by 84.4%, achieving an average bit-width of as low as 5.1 bits, compared with existing solutions. Moreover, STQuant incurs only O(N/K) computational overhead and requires O(1) extra space.