Improved stochastic rounding

TL;DR

This paper improves stochastic rounding by analyzing its properties, proposing new distributions to balance variance and bias, and validating these through simulations on various operations in low-precision computations.

Contribution

It introduces a theoretical analysis of SR, proposes new probability distributions to optimize variance and bias trade-offs, and validates these methods through comprehensive simulations.

Findings

Upper bound of rounding variance established and validated

New SR distributions effectively balance variance and bias

Simulation results demonstrate improved SR performance in low-precision operations

Abstract

Due to the limited number of bits in floating-point or fixed-point arithmetic, rounding is a necessary step in many computations. Although rounding methods can be tailored for different applications, round-off errors are generally unavoidable. When a sequence of computations is implemented, round-off errors may be magnified or accumulated. The magnification of round-off errors may cause serious failures. Stochastic rounding (SR) was introduced as an unbiased rounding method, which is widely employed in, for instance, the training of neural networks (NNs), showing a promising training result even in low-precision computations. Although the employment of SR in training NNs is consistently increasing, the error analysis of SR is still to be improved. Additionally, the unbiased rounding results of SR are always accompanied by large variances. In this study, some general properties of SR are stated and proven. Furthermore, an upper bound of rounding variance is introduced and validated. Two new probability distributions of SR are proposed to study the trade-off between variance and bias, by solving a multiple objective optimization problem. In the simulation study, the rounding variance, bias, and relative errors of SR are studied for different operations, such as summation, square root calculation through Newton iteration and inner product computation, with specific rounding precision.