CUDAHercules: Benchmarking Hardware-Aware Expert-level CUDA Optimization for LLMs

TL;DR

CUDAHercules is a comprehensive benchmark that assesses the ability of AI models to generate expert-level CUDA code for large language models, revealing significant gaps in current AI capabilities.

Contribution

Introduces CUDAHercules, a benchmark for evaluating AI-generated CUDA code against expert standards across multiple hardware architectures and tasks.

Findings

Models often compile and pass tests but lack expert optimization strategies.

Semantic understanding reduces success rates in CUDA code generation.

Feedback and tools can improve correctness but lead to slower fallback solutions.

Abstract

Large language models show promise for automated CUDA programming, however even the strongest coding models (e.g., Claude-Opus-4.6) may still fall short of expert-level, architecture-aware optimization. We introduce CUDAHercules, a benchmark that evaluates generated CUDA against end-to-end human-expert SOTA systems. It spans single kernels, module-level operators, full applications, and unsolved challenge tasks across Ampere, Hopper, and Blackwell GPUs, with end-to-end tasks gated by domain-specific semantic validators. Evaluating models such as Claude-Opus-4.6 and GPT-5.4 shows a large gap between runnable CUDA and expert CUDA engineering: models often compile and pass tests, but rarely recover the optimization strategies needed to match expert performance. Application semantics further reduce success, and iterative or tool-augmented feedback can improve correctness while drifting toward slow fallback implementations. These results show that automated CUDA programming remains far from fully solved and requires stronger hardware reasoning, better tool use, and training objectives that connect code understanding to hardware architecture-grounded intelligence.