Counting Without Running: Evaluating LLMs' Reasoning About Code Complexity

TL;DR

This paper introduces gpuFLOPBench, a benchmark for evaluating LLMs' ability to predict GPU kernel FLOP counts, revealing current models' limitations in reasoning about implicit hardware-specific behaviors.

Contribution

It presents gpuFLOPBench, a novel benchmark for assessing LLMs' reasoning about GPU code complexity and performance prediction.

Findings

Current LLMs excel at simple FLOP counting but struggle with implicit behaviors.

Implicit FLOP prediction errors can be several orders of magnitude.

Benchmark highlights core limitations in LLM reasoning about hardware-specific effects.

Abstract

Modern GPU software stacks demand developers who can anticipate performance bottlenecks before ever launching a kernel; misjudging floating-point workloads upstream can derail tuning, scheduling, and even hardware procurement. Yet despite rapid progress in code generation, today's Large Language Models (LLMs) are rarely tested on this kind of forward-looking reasoning. We close that gap with gpuFLOPBench, a benchmark that asks models to "count without running" by predicting single and double-precision FLOP counts for 577 CUDA kernels drawn from HeCBench, annotated with ground-truth profiles and eight execution attributes that distinguish trivially analyzable code from kernels whose FLOPs depend on hidden compiler or runtime behavior. Evaluating current closed-source reasoning models shows clear but uneven progress: the newest LLMs achieve perfect classification on straightforward kernels but still incur multiple order-of-magnitude errors whenever implicit FLOPs arise from division, intrinsic math functions, or common subexpressions. These results surface a core limitation of existing code assistants -- the inability to internalize hardware-specific microcode effects -- and position gpuFLOPBench as a focused testbed for developing LLM tooling that can reason about performance with the same rigor as experienced GPU developers. Sources are available at our repository: https://github.com/Scientific-Computing-Lab/gpuFLOPBench