The Positive Effects of Stochastic Rounding in Numerical Algorithms

TL;DR

This paper investigates stochastic rounding's theoretical advantages in numerical algorithms, demonstrating its unbiased nature and providing probabilistic error bounds that outperform traditional deterministic bounds, thus supporting wider adoption in computer architecture.

Contribution

It extends theoretical analysis of stochastic rounding, comparing bias in different modes and establishing probabilistic error bounds for polynomial evaluation.

Findings

SR-nearness is unbiased, unlike SR-up-or-down.

Probabilistic error bound for Horner's method is O(√n), better than deterministic O(n).

SR can improve error control in numerical algorithms.

Abstract

Recently, stochastic rounding (SR) has been implemented in specialized hardware but most current computing nodes do not yet support this rounding mode. Several works empirically illustrate the benefit of stochastic rounding in various fields such as neural networks and ordinary differential equations. For some algorithms, such as summation, inner product or matrixvector multiplication, it has been proved that SR provides probabilistic error bounds better than the traditional deterministic bounds. In this paper, we extend this theoretical ground for a wider adoption of SR in computer architecture. First, we analyze the biases of the two SR modes: SR-nearness and SR-up-or-down. We demonstrate on a case-study of Euler's forward method that IEEE-754 default rounding modes and SR-up-or-down accumulate rounding errors across iterations and that SR-nearness, being unbiased, does not. Second, we prove a O($\sqrt$ n) probabilistic bound on the forward error of Horner's polynomial evaluation method with SR, improving on the known deterministic O(n) bound.