CuBridge: An LLM-Based Framework for Understanding and Reconstructing High-Performance Attention Kernels

TL;DR

CuBridge is an LLM-based framework that transforms expert CUDA attention kernels into optimized, correct GPU code from PyTorch specifications, outperforming existing methods across variants and platforms.

Contribution

It introduces a structured lift-transfer-lower workflow that adapts expert kernels into high-performance, correct CUDA implementations from high-level specifications.

Findings

CuBridge consistently produces correct kernels across diverse attention variants.

It substantially outperforms general frameworks, compiler-based approaches, and prior LLM methods.

The framework works effectively across multiple GPU platforms.

Abstract

Efficient CUDA implementations of attention mechanisms are critical to modern deep learning systems, yet supporting diverse and evolving attention variants remains challenging. Existing frameworks and compilers trade performance for flexibility, while expert-written kernels achieve high efficiency but are difficult to adapt. Recent work explores large language models (LLMs) for GPU kernel generation, but prior studies report unstable correctness and significant performance gaps for complex operators such as attention. We present CuBridge, an LLM-based framework that adapts expert-written attention kernels through a structured lift-transfer-lower workflow. CuBridge starts from expert-written CUDA attention kernels and lifts them into an executable intermediate representation that makes execution orchestration explicit while abstracting low-level CUDA syntax. Given a user-provided PyTorch specification, CuBridge generates and verifies a target IR program, then reconstructs optimized CUDA code via reference-guided lowering. Across diverse attention variants and GPU platforms, CuBridge consistently produces correct kernels and substantially outperforms general frameworks, compiler-based approaches, and prior LLM-based methods.