CUDABeaver: Benchmarking LLM-Based Automated CUDA Debugging

TL;DR

CUDABeaver introduces a benchmark and evaluation protocol for assessing the effectiveness of LLM-based CUDA debugging tools, emphasizing the importance of performance preservation and realistic failure scenarios.

Contribution

The paper presents CUDABeaver, a new benchmark with a protocol-conditional metric for more accurate evaluation of LLM-based CUDA debugging methods.

Findings

Protocol-aware evaluation reveals significant performance sensitivity in CUDA fixers.

When allowing high performance loss, fixers show much higher success rates.

Stricter performance requirements sharply reduce the measured success of debugging tools.

Abstract

Debugging CUDA programs has long been challenging because failures often arise from subtle interactions among hardware behavior, compiler decisions, memory hierarchy, and asynchronous execution. More importantly, with the rapid expansion of GPU usage across scientific computing, machine learning, graphics, and systems workloads, CUDA debugging has become more challenging than ever. Current evaluations of LLM-based CUDA programming largely miss this setting: a model can pass correctness tests with repair by degeneration, simplifying the CUDA code into a safer but slower program that abandons the original optimization structure. We introduce CUDABEAVER, a benchmark for CUDA debugging from real failing workspaces produced during LLM-based CUDA generation. Each task provides the broken candidate, native build/test commands, raw error evidence, and a single editable file. CUDABEAVER evaluates whether a fixer truly repairs the failing CUDA code or merely finds a slower test-passing replacement, reporting results by failure category, debugging trajectory, stagnation mode, and performance preservation. We further propose pass@k(M,C,A), a protocol-conditional CUDA debugging metric by making the fixer M, corpus C, and protocol axes Aexplicit. Using this metric across 213 tasks and seven frontier LLMs, we show that protocol-aware evaluation gives a more faithful view of CUDA debugging ability: when performance-loss tolerance is high, fixers appear much stronger, but even a minor stricter performance requirement can sharply reduce measured success, shifting scores by up to 40 percentage points.