On the influence of stochastic roundoff errors and their bias on the convergence of the gradient descent method with low-precision floating-point computation

TL;DR

This paper investigates how stochastic rounding errors, especially biased ones, influence the convergence of gradient descent in low-precision floating-point computations, proposing new schemes that improve convergence.

Contribution

It introduces two novel stochastic rounding schemes with controlled bias that enhance convergence in low-precision gradient descent, supported by theoretical analysis and empirical validation.

Findings

Biased rounding can still promote convergence in low-precision settings.

Proposed schemes outperform unbiased rounding in certain convex optimization tasks.

Empirical results confirm improved training stability with new rounding methods.

Abstract

When implementing the gradient descent method in low precision, the employment of stochastic rounding schemes helps to prevent stagnation of convergence caused by the vanishing gradient effect. Unbiased stochastic rounding yields zero bias by preserving small updates with probabilities proportional to their relative magnitudes. This study provides a theoretical explanation for the stagnation of the gradient descent method in low-precision computation. Additionally, we propose two new stochastic rounding schemes that trade the zero bias property with a larger probability to preserve small gradients. Our methods yield a constant rounding bias that, on average, lies in a descent direction. For convex problems, we prove that the proposed rounding methods typically have a beneficial effect on the convergence rate of gradient descent. We validate our theoretical analysis by comparing the performances of various rounding schemes when optimizing a multinomial logistic regression model and when training a simple neural network with an 8-bit floating-point format.