Spiking Transformer Hardware Accelerators in 3D Integration

TL;DR

This paper introduces a novel 3D hardware architecture for spiking transformers, optimizing energy efficiency and delay through co-design and 3D integration techniques, addressing the lack of dedicated hardware support.

Contribution

It presents the first 3D hardware architecture and design methodology specifically tailored for spiking transformers, leveraging 3D integration for performance gains.

Findings

Significant energy reduction compared to 2D CMOS implementations

Delay improvements achieved through 3D stacking

Effective architecture and physical design co-optimization

Abstract

Spiking neural networks (SNNs) are powerful models of spatiotemporal computation and are well suited for deployment on resource-constrained edge devices and neuromorphic hardware due to their low power consumption. Leveraging attention mechanisms similar to those found in their artificial neural network counterparts, recently emerged spiking transformers have showcased promising performance and efficiency by capitalizing on the binary nature of spiking operations. Recognizing the current lack of dedicated hardware support for spiking transformers, this paper presents the first work on 3D spiking transformer hardware architecture and design methodology. We present an architecture and physical design co-optimization approach tailored specifically for spiking transformers. Through memory-on-logic and logic-on-logic stacking enabled by 3D integration, we demonstrate significant energy and delay improvements compared to conventional 2D CMOS integration.