Deep Kernel Fusion for Transformers

TL;DR

DeepFusionKernel is a novel deeply fused kernel designed to reduce memory bandwidth bottlenecks in agentic LLM inference, significantly improving speed on modern GPUs by optimizing cache reuse and HBM traffic.

Contribution

It introduces DeepFusionKernel, a new kernel that enhances LLM inference efficiency by reducing memory traffic and improving cache reuse, adaptable across models and hardware.

Findings

Up to 13.2% speedup on H100 GPU

Up to 9.7% speedup on A100 GPU

Consistent acceleration across various models and inference lengths

Abstract

Agentic LLM inference with long contexts is increasingly limited by memory bandwidth rather than compute. In this setting, SwiGLU MLP blocks, whose large weights exceed cache capacity, become a major yet under-optimized bottleneck. We propose DeepFusionKernel, a deeply fused kernel that cuts HBM traffic and boosts cache reuse, delivering up to 13.2% speedup on H100 and 9.7% on A100 over SGLang. Integrated with SGLang and paired with a kernel scheduler, DeepFusionKernel ensures consistent accelerations over generation lengths, while remaining adaptable to diverse models, inference configurations, and hardware platforms.