The Anatomy of a Triton Attention Kernel

TL;DR

This paper presents a portable, high-performance attention kernel for large language model inference that works efficiently across NVIDIA and AMD GPUs using Triton, demonstrating significant performance improvements and model portability.

Contribution

The authors develop a state-of-the-art, cross-platform attention kernel using Triton, achieving high efficiency and portability across different GPU architectures with systematic tuning and integration.

Findings

Achieves 105.9% of state-of-the-art performance on inference tasks.

Demonstrates effective cross-vendor GPU portability.

Highlights the potential of domain-specific languages for model deployment.

Abstract

A long-standing goal in both industry and academia is to develop an LLM inference platform that is portable across hardware architectures, eliminates the need for low-level hand-tuning, and still delivers best-in-class efficiency. In this work, we demonstrate that portable, efficient cross-platform LLM inference is indeed possible and share our experience. We develop a state-of-the-art paged attention kernel, the core performance-critical component of many LLM deployments, that builds exclusively on the domain-specific just-in-time compiled language Triton to achieve state-of-the-art performance on both NVIDIA and AMD GPUs. We describe our high-level approach, the key algorithmic and system-level improvements, the parameter auto-tuning required to unlock efficiency, and the integrations into a popular inference server that are necessary to bring the performance of a generic Triton attention kernel from 19.7% of the state-of-the-art to 105.9%. Our results highlight how open-source domain-specific languages can be leveraged to unlock model portability across different GPU vendors.