Attention in SRAM on Tenstorrent Grayskull

TL;DR

This paper demonstrates how utilizing SRAM in the Tenstorrent Grayskull architecture for Transformer attention operations significantly speeds up computation, especially for Softmax, and compares its cost-effectiveness to GPUs.

Contribution

It introduces a fused kernel that maximizes SRAM use for attention, including a dedicated Softmax kernel and a CPU baseline, achieving notable speedups.

Findings

Softmax kernel speedup up to 10x over CPU

Fused kernel is 1.8x faster than dedicated Softmax

Grayskull is 30x cheaper than Nvidia H100 and has 1.5x more SRAM

Abstract

When implementations of the Transformer's self-attention layer utilize SRAM instead of DRAM, they can achieve significant speedups. The Tenstorrent Grayskull architecture provides a large SRAM, distributed across a grid of cores. This work presents a fused kernel for Grayskull, that exclusively utilizes its large SRAM by combining matrix multiplication, attention score scaling and Softmax operations. Additionally, a dedicated Softmax kernel utilizing the SRAM and a CPU implementation serving as a baseline are presented. The Softmax operation consumes most of the runtime in the computation of attention weights from queries and keys on Grayskull. The speedup of the dedicated Softmax kernel compared to the CPU implementation is up to $10 \times$, and the Softmax implementation inside the fused kernel is approximately $1.8 \times$ faster than the dedicated Softmax kernel. The time and memory complexity of all implementations is quadratic in sequence length. Currently, the Grayskull e150 is approximately $30 \times$ cheaper for the general public than an Nvidia H100 PCIe (a state-of-the-art GPU) and offers approximately $1.5 \times$ more SRAM.