Kernel Looping: Eliminating Synchronization Boundaries for Peak Inference Performance

TL;DR

Kernel looping is a novel optimization technique that reduces synchronization overheads in GPU-based AI inference, significantly boosting token generation speed and hardware utilization.

Contribution

This paper introduces kernel looping, a specialized global optimization that eliminates synchronization costs between kernel calls in modern dataflow architectures for AI inference.

Findings

Speeds up decode phase by up to 2.2× on SN40L

Achieves up to 2.5× speedup across multiple sockets

Enables over 90% peak performance on 8 and 16 sockets

Abstract

Token generation speed is critical to power the next wave of AI inference applications. GPUs significantly underperform during token generation due to synchronization overheads at kernel boundaries, utilizing only 21% of their peak memory bandwidth. While recent dataflow architectures mitigate these overheads by enabling aggressive fusion of decoder layers into a single kernel, they too leave performance on the table due to synchronization penalties at layer boundaries. This paper presents kernel looping, a specialized global optimization technique which exploits an optimization opportunity brought by combining the unique layer-level fusion possible in modern dataflow architectures with the repeated layer structure found in language models. Kernel looping eliminates synchronization costs between consecutive calls to the same kernel by transforming these calls into a single call to a modified kernel containing a pipelined outer loop. We evaluate kernel looping on the SambaNova SN40L Reconfigurable Dataflow Unit (RDU), a commercial dataflow accelerator for AI. Experiments demonstrate that kernel looping speeds up the decode phase of a wide array of powerful open-source models by up to 2.2$\times$ on SN40L. Kernel looping allows scaling of decode performance over multiple SN40L sockets, achieving speedups of up to 2.5$\times$. Finally, kernel looping enables SN40L to achieve over 90% of peak performance on 8 and 16 sockets and achieve a speedup of up to 3.7$\times$ over DGX H100. Kernel looping, as well as the models evaluated in this paper, are deployed in production in a commercial AI inference cloud.