A Metric Driven Approach to Mixed Precision Training

TL;DR

This paper introduces a metric-driven methodology for selecting mixed precision numerics to optimize deep learning training efficiency, demonstrated on language models and applicable to various architectures.

Contribution

It proposes a novel metric-based approach to guide mixed precision training, enhancing scalability and efficiency in deep learning models.

Findings

Effective scaling of language models using the proposed metric-driven approach

Generalizable methodology applicable to different neural network architectures

Improved training efficiency with mixed precision numerics

Abstract

As deep learning methodologies have developed, it has been generally agreed that increasing neural network size improves model quality. However, this is at the expense of memory and compute requirements, which also need to be increased. Various efficiency techniques have been proposed to rein in hardware costs, one being the use of low precision numerics. Recent accelerators have introduced several different 8-bit data types to help accommodate DNNs in terms of numerics. In this paper, we identify a metric driven methodology to aid in the choice of numerics. We demonstrate how such a methodology can help scale training of a language representation model. The technique can be generalized to other model architectures.