Probabilistic Error Analysis For Sequential Summation of Real Floating Point Numbers

TL;DR

This paper develops probabilistic bounds for the error in summing real floating point numbers, showing they can be significantly tighter than deterministic bounds through experiments with large data sets.

Contribution

It introduces two probabilistic bounds based on Azuma's inequalities for floating point summation errors, improving error estimation accuracy.

Findings

Probabilistic bounds are 1-2 orders tighter than deterministic bounds.

Bounds are tighter when summands have the same sign.

Experiments with up to 10^7 numbers demonstrate the bounds' effectiveness.

Abstract

We derive two probabilistic bounds for the relative forward error in the floating point summation of $n$ real numbers, by representing the roundoffs as independent, zero-mean, bounded random variables. The first probabilistic bound is based on Azuma's concentration inequality, and the second on the Azuma-Hoeffding Martingale. Our numerical experiments illustrate that the probabilistic bounds, with a stringent failure probability of $10^{-16}$, can be 1-2 orders of magnitude tighter than deterministic bounds. We performed the numerical experiments in Julia by summing up to $n=10^7$ single precision (binary32) floating point numbers, and up to $n=10^4$ half precision (binary16) floating point numbers. We simulated exact computation with double precision (binary64). The bounds tend to be tighter when all summands have the same sign.