CUDABench: Benchmarking LLMs for Text-to-CUDA Generation

TL;DR

CUDABench is a comprehensive benchmark for evaluating the ability of large language models to generate CUDA code from text, addressing correctness, functional accuracy, and performance across diverse application domains.

Contribution

This work introduces CUDABench, a novel benchmark with evaluation metrics and verification pipelines specifically designed for assessing text-to-CUDA generation by LLMs.

Findings

High compilation success does not guarantee functional correctness

LLMs lack domain-specific algorithmic knowledge

Generated code often underutilizes GPU hardware resources

Abstract

Recent studies have demonstrated the potential of Large Language Models (LLMs) in generating GPU Kernels. Current benchmarks focus on the translation of high-level languages into CUDA, overlooking the more general and challenging task of text-to-CUDA generation. Furthermore, given the hardware-specific and performance-critical features of GPU programming, accurately assessing the performance of LLM-generated GPU programs is nontrivial. In this work, we introduce CUDABench, a comprehensive benchmark designed to evaluate the text-to-CUDA capabilities of LLMs. First, we construct CUDABench-Set, which covers Breadth-Depth-Difficulty evaluation space in diverse application domains, including artificial intelligence, scientific computing, and data analytics, etc. Furthermore, we propose CUDABench-Score and Generative Verification Pipeline that assess (1) compilation correctness, (2) functional consistency through execution-based verification, and (3) a novel roofline-based metric, Performance-Score. Benchmarking state-of-the-art LLMs reveals insightful findings and challenges of text-to-CUDA, such as a notable mismatch between high compilation success rates and low functional correctness, a lack of domain-specific algorithmic knowledge, and suboptimal utilization of GPU hardware resources. Our benchmark is available at https://github.com/CUDA-Bench/CUDABench.