Memory Efficient Mixed-Precision Optimizers

TL;DR

This paper introduces memory-efficient mixed-precision optimizers that reduce memory usage and training time by eliminating floating point copies and integrating optimizer steps during back-propagation, without sacrificing accuracy.

Contribution

It presents a novel algorithm that minimizes memory during training by removing floating point copies and executing optimizer steps during back-propagation.

Findings

Up to 25% reduction in peak memory usage

15% faster training times

Maintains the same model accuracy

Abstract

Traditional optimization methods rely on the use of single-precision floating point arithmetic, which can be costly in terms of memory size and computing power. However, mixed precision optimization techniques leverage the use of both single and half-precision floating point arithmetic to reduce memory requirements while maintaining model accuracy. We provide here an algorithm to further reduce memory usage during the training of a model by getting rid of the floating point copy of the parameters, virtually keeping only half-precision numbers. We also explore the benefits of getting rid of the gradient's value by executing the optimizer step during the back-propagation. In practice, we achieve up to 25% lower peak memory use and 15% faster training while maintaining the same level of accuracy.