Beyond Application End-Point Results: Quantifying Statistical Robustness of MCMC Accelerators

TL;DR

This paper introduces a comprehensive framework for evaluating the statistical robustness of MCMC accelerators, emphasizing metrics beyond mere accuracy to ensure reliable probabilistic computations in hardware designs.

Contribution

It proposes a novel methodology with three key metrics for assessing the correctness of probabilistic hardware accelerators, addressing gaps in current evaluation practices.

Findings

Statistical robustness can match software quality with minimal hardware changes.

The framework exposes design issues not visible through accuracy alone.

Guides hardware design to balance performance and statistical correctness.

Abstract

Statistical machine learning often uses probabilistic algorithms, such as Markov Chain Monte Carlo (MCMC), to solve a wide range of problems. Probabilistic computations, often considered too slow on conventional processors, can be accelerated with specialized hardware by exploiting parallelism and optimizing the design using various approximation techniques. Current methodologies for evaluating correctness of probabilistic accelerators are often incomplete, mostly focusing only on end-point result quality ("accuracy"). It is important for hardware designers and domain experts to look beyond end-point "accuracy" and be aware of the hardware optimizations impact on other statistical properties. This work takes a first step towards defining metrics and a methodology for quantitatively evaluating correctness of probabilistic accelerators beyond end-point result quality. We propose three pillars of statistical robustness: 1) sampling quality, 2) convergence diagnostic, and 3) goodness of fit. We apply our framework to a representative MCMC accelerator and surface design issues that cannot be exposed using only application end-point result quality. Applying the framework to guide design space exploration shows that statistical robustness comparable to floating-point software can be achieved by slightly increasing the bit representation, without floating-point hardware requirements.