A Survey of Numerical Methods Utilizing Mixed Precision Arithmetic

TL;DR

This survey reviews the recent developments in mixed and multiprecision arithmetic, emphasizing hardware advancements, software capabilities, and their impact on numerical linear algebra performance improvements.

Contribution

It provides a comprehensive landscape analysis of existing multiprecision methods, current efforts, and potential for future production-quality solutions.

Findings

Mixed precision can significantly improve performance in numerical linear algebra.

Current hardware supports specialized low-precision units like NVIDIA tensor cores.

Preliminary results show application outperforming traditional fixed precision methods.

Abstract

Within the past years, hardware vendors have started designing low precision special function units in response to the demand of the Machine Learning community and their demand for high compute power in low precision formats. Also the server-line products are increasingly featuring low-precision special function units, such as the NVIDIA tensor cores in ORNL's Summit supercomputer providing more than an order of magnitude higher performance than what is available in IEEE double precision. At the same time, the gap between the compute power on the one hand and the memory bandwidth on the other hand keeps increasing, making data access and communication prohibitively expensive compared to arithmetic operations. To start the multiprecision focus effort, we survey the numerical linear algebra community and summarize all existing multiprecision knowledge, expertise, and software capabilities in this landscape analysis report. We also include current efforts and preliminary results that may not yet be considered "mature technology," but have the potential to grow into production quality within the multiprecision focus effort. As we expect the reader to be familiar with the basics of numerical linear algebra, we refrain from providing a detailed background on the algorithms themselves but focus on how mixed- and multiprecision technology can help improving the performance of these methods and present highlights of application significantly outperforming the traditional fixed precision methods.