FireFly-T: High-Throughput Sparsity Exploitation for Spiking Transformer Acceleration with Dual-Engine Overlay Architecture

TL;DR

FireFly-T is a novel dual-engine overlay architecture that significantly accelerates spiking transformer computations by exploiting sparsity and optimizing dataflows, achieving substantial improvements in energy and DSP efficiency.

Contribution

The paper introduces FireFly-T, a dual-engine overlay architecture with a sparse engine and a binary engine, enabling scalable, high-throughput acceleration of spiking transformers on FPGA hardware.

Findings

Achieves 1.39x and 2.40x higher energy efficiency compared to prior accelerators.

Attains 4.21x and 7.10x greater DSP efficiency over previous designs.

Demonstrates effective exploitation of fine-grained sparsity and scalable parallelism.

Abstract

Spiking transformers are emerging as a promising architecture that combines the energy efficiency of Spiking Neural Networks (SNNs) with the powerful attention mechanisms of transformers. However, existing hardware accelerators lack support for spiking attention, exhibit limited throughput in exploiting fine-grained sparsity, and struggle with scalable parallelism in sparse computation. To address these, we propose FireFly-T, a dual-engine overlay architecture that integrates a sparse engine for activation sparsity and a binary engine for spiking attention. In the sparse engine, we propose a highthroughput sparse decoder that exploits fine-grained sparsity by concurrently extracting multiple non-zero spikes. To complement this, we introduce a scalable load balancing mechanism with weight dispatch and out-of-order execution, eliminating bank conflicts to support scalable multidimensional parallelism. In the binary engine, we leverage the byte-level write capability of SRAMs to efficiently manipulate the 3D dataflows required for spiking attention with minimal resource overhead. We also optimize the core AND-PopCount operation in spiking attention through a LUT6-based implementation, improving timing closure and reducing LUT utilization on Xilinx FPGAs. As an overlay architecture, FireFly-T further incorporates an orchestrator that dynamically manipulates input dataflows with flexible adaptation for diverse network topologies, while ensuring efficient resource utilization and maintaining high throughput. Experimental results demonstrate that our accelerator achieves $1.39\times$ and $2.40\times$ higher energy efficiency, as well as $4.21\times$ and $7.10\times$ greater DSP efficiency, compared to FireFly v2 and the transformer-enabled SpikeTA, respectively. These results highlight its potential as an efficient hardware platform for spiking transformer.