Improving deep neural network performance through sampling

TL;DR

This paper explores energy-efficient sampling methods using probabilistic neurons to enhance deep neural network accuracy, comparing sampling versus increasing bit precision.

Contribution

It demonstrates the feasibility of using multiple samples from probabilistic networks to improve accuracy and provides an energy tradeoff analysis between sampling and bit precision.

Findings

Probabilistic sampling can yield superior accuracy in deep neural networks.

A simple expression estimates energy tradeoffs between sampling and bit precision.

Results illustrate the energy efficiency of probabilistic sampling over deterministic approaches.

Abstract

Energy efficient sampling with probabilistic neurons or p-bits has been demonstrated in the context of Boltzmann machines and it is natural to ask if these approaches can be extended to the field of generative AI where energy costs have become prohibitively large. However, this very active field is dominated by feedforward deep neural networks (DNNs) which primarily use multi-bit deterministic neurons with no role for sampling. In this paper we first show that it is feasible to obtain superior accuracy through the use of multiple samples generated by probabilistic networks. This possibility raises the question of which option is energetically preferable for improving accuracy: generating more samples, or adding more bits to a single deterministic sample. We provide a simple expression that can be used to estimate these energy tradeoffs and illustrate it with results for different algorithms and architectures.