Next-generation Probabilistic Computing Hardware with 3D MOSAICs, Illusion Scale-up, and Co-design

TL;DR

This paper proposes a new hardware-centric approach to probabilistic computing, emphasizing 3D MOSAIC integration and a scalable distributed framework called Illusion, aiming to advance probabilistic algorithms beyond current deterministic AI hardware.

Contribution

It introduces the concept of 3D MOSAIC hardware integration and the Illusion distributed framework to enhance probabilistic computing hardware and scalability.

Findings

Highlights challenges in probabilistic hardware implementation.

Proposes 3D MOSAIC integration for improved performance.

Introduces Illusion framework for scalable probabilistic computing.

Abstract

The vast majority of 21st century AI workloads are based on gradient-based deterministic algorithms such as backpropagation. One of the key reasons for the dominance of deterministic ML algorithms is the emergence of powerful hardware accelerators (GPU and TPU) that have enabled the wide-scale adoption and implementation of these algorithms. Meanwhile, discrete and probabilistic Monte Carlo algorithms have long been recognized as one of the most successful algorithms in all of computing with a wide range of applications. Specifically, Markov Chain Monte Carlo (MCMC) algorithm families have emerged as the most widely used and effective method for discrete combinatorial optimization and probabilistic sampling problems. We adopt a hardware-centric perspective on probabilistic computing, outlining the challenges and potential future directions to advance this field. We identify two critical research areas: 3D integration using MOSAICs (Monolithic/Stacked/Assembled ICs) and the concept of Illusion, a hardware-agnostic distributed computing framework designed to scale probabilistic accelerators.