Dither computing: a hybrid deterministic-stochastic computing framework

TL;DR

Dither computing is a hybrid framework that combines stochastic and deterministic methods to perform unbiased arithmetic with optimal mean squared error, improving upon traditional stochastic and deterministic approaches.

Contribution

It introduces dither computing, a novel hybrid approach that achieves unbiasedness and optimal MSE order, enhancing stochastic rounding and arithmetic accuracy.

Findings

Dither computing achieves unbiased estimates with MSE of order 1/N^2.

Experimental results show improved accuracy over traditional stochastic and deterministic methods.

The framework benefits deep learning applications involving stochastic rounding.

Abstract

Stochastic computing has a long history as an alternative method of performing arithmetic on a computer. While it can be considered an unbiased estimator of real numbers, it has a variance and MSE on the order of $\Omega(\frac{1}{N})$. On the other hand, deterministic variants of stochastic computing remove the stochastic aspect, but cannot approximate arbitrary real numbers with arbitrary precision and are biased estimators. However, they have an asymptotically superior MSE on the order of $O(\frac{1}{N^2})$. Recent results in deep learning with stochastic rounding suggest that the bias in the rounding can degrade performance. We proposed an alternative framework, called dither computing, that combines aspects of stochastic computing and its deterministic variants and that can perform computing with similar efficiency, is unbiased, and with a variance and MSE also on the optimal order of $\Theta(\frac{1}{N^2})$. We also show that it can be beneficial in stochastic rounding applications as well. We provide implementation details and give experimental results to comparatively show the benefits of the proposed scheme.