Mixed precision in Graphics Processing Unit

TL;DR

This paper reviews the use of mixed precision in GPUs, highlighting its benefits for energy efficiency and machine learning, and discusses hardware implementations like NVIDIA Tensor Cores and phase change memories.

Contribution

It provides a comprehensive overview of mixed precision applications, standards, and hardware implementations, emphasizing its advantages and challenges in modern GPU computing.

Findings

Mixed precision can improve neural network training efficiency.

Hardware like NVIDIA Tensor Cores enables effective mixed precision computation.

Abandoning traditional models allows for innovative memory-based mixed precision approaches.

Abstract

Modern graphics computing units (GPUs) are designed and optimized to perform highly parallel numerical calculations. This parallelism has enabled (and promises) significant advantages, both in terms of energy performance and calculation. In this document, we take stock of the different applications of mixed precision. We recall the standards currently used in the overwhelming majority of systems in terms of numerical computation. We show that the mixed precision which decreases the precision at the input of an operation does not necessarily decrease the precision of its output. We show that this previous principle allows its transposition into one of the branches that most needs computing power: machine learning. The use of fixed point numbers and half-precision are two very effective ways to increase the learning ability of complex neural networks. Mixed precision still requires the use of suitable hardware, failing which the calculation time could on the contrary be lengthened. The NVIDIA Tensor Core that is found among others in their Tesla V100 range, is an example of implementation at the hardware level of mixed precision. On the other hand, by abandoning the traditional von Neumann model, mixed precision can also be transposed to a lower level of abstraction, using phase change memories.