$\mu$nit Scaling: Simple and Scalable FP8 LLM Training

TL;DR

This paper introduces $;nit Scaling, a simple and scalable FP8 training method for large language models that eliminates the need for dynamic scaling or hyperparameter tuning, enabling faster training with maintained quality.

Contribution

The paper presents $;nit Scaling, a novel FP8 training approach that simplifies implementation, removes the need for dynamic scaling, and allows hyperparameter transfer across model sizes.

Findings

Achieves comparable quality to higher precision training.

Enables up to 33% faster training.

Supports training models from 1B to 13B parameters.

Abstract

Large Language Model training with 8-bit floating point (FP8) formats promises significant efficiency improvements, but reduced numerical precision makes training challenging. It is currently possible to train in FP8 only if one is willing to tune various hyperparameters, reduce model scale, or accept the overhead of computing dynamic scale factors. We demonstrate simple, scalable FP8 training that requires no dynamic scaling factors or special hyperparameters, even at large model sizes. Our method, $\mu$nit Scaling ($\mu$S), also enables simple hyperparameter transfer across model widths, matched numerics across training and inference, and other desirable properties. $\mu$nit Scaling is straightforward to implement, consisting of a set of minimal interventions based on a first-principles analysis of common transformer operations. We validate our method by training models from 1B to 13B parameters, performing all hidden linear layer computations in FP8. We achieve quality equal to higher precision baselines while also training up to 33% faster.