cuPilot: A Strategy-Coordinated Multi-agent Framework for CUDA Kernel Evolution

TL;DR

cuPilot is a multi-agent framework that uses strategy-based representations and advanced algorithms to automatically optimize CUDA kernels, achieving significant speedups and hardware utilization improvements.

Contribution

It introduces a novel strategy-coordinated evolution approach with roofline-guided prompting and population initialization for kernel optimization.

Findings

Achieves an average 3.09× speedup over PyTorch on 100 kernels.

Demonstrates high hardware utilization on GEMM tasks.

Produces open-source optimized kernels.

Abstract

Optimizing CUDA kernels is a challenging and labor-intensive task, given the need for hardware-software co-design expertise and the proprietary nature of high-performance kernel libraries. While recent large language models (LLMs) combined with evolutionary algorithms show promise in automatic kernel optimization, existing approaches often fall short in performance due to their suboptimal agent designs and mismatched evolution representations. This work identifies these mismatches and proposes cuPilot, a strategy-coordinated multi-agent framework that introduces strategy as an intermediate semantic representation for kernel evolution. Key contributions include a strategy-coordinated evolution algorithm, roofline-guided prompting, and strategy-level population initialization. Experimental results show that the generated kernels by cuPilot achieve an average speed up of 3.09$\times$ over PyTorch on a benchmark of 100 kernels. On the GEMM tasks, cuPilot showcases sophisticated optimizations and achieves high utilization of critical hardware units. The generated kernels are open-sourced at https://github.com/champloo2878/cuPilot-Kernels.git.