Configurable p-Neurons Using Modular p-Bits

TL;DR

This paper introduces a modular p-bit design that enables configurable probabilistic activation functions for neural networks, demonstrated through spintronic and FPGA implementations, offering significant hardware resource savings.

Contribution

The paper re-engineers p-bits into modular units allowing a range of probabilistic activation functions, including sigmoid, Tanh, and ReLU, with practical spintronic and FPGA implementations.

Findings

Wide and tunable probabilistic ranges achieved.

Order of magnitude hardware resource savings demonstrated.

Successful FPGA implementation with shared stochastic units.

Abstract

Probabilistic bits (p-bits) have recently been employed in neural networks (NNs) as stochastic neurons with sigmoidal probabilistic activation functions. Nonetheless, there remain a wealth of other probabilistic activation functions that are yet to be explored. Here we re-engineer the p-bit by decoupling its stochastic signal path from its input data path, giving rise to a modular p-bit that enables the realization of probabilistic neurons (p-neurons) with a range of configurable probabilistic activation functions, including a probabilistic version of the widely used Logistic Sigmoid, Tanh and Rectified Linear Unit (ReLU) activation functions. We present spintronic (CMOS + sMTJ) designs that show wide and tunable probabilistic ranges of operation. Finally, we experimentally implement digital-CMOS versions on an FPGA, with stochastic unit sharing, and demonstrate an order of magnitude (10x) saving in required hardware resources compared to conventional digital p-bit implementations.