Hunting CUDA Bugs at Scale with cuFuzz

TL;DR

cuFuzz is a novel CUDA fuzzing tool that effectively uncovers memory-safety and concurrency bugs in GPU programs by addressing key obstacles in GPU fuzz testing, significantly improving bug detection and coverage.

Contribution

This work introduces cuFuzz, the first practical CUDA-oriented fuzzer that overcomes kernel-level false positives, integrates device-side coverage feedback, and separates sanitization from coverage collection.

Findings

Uncovered 43 previously unknown bugs, including memory access violations and data races.

Achieved higher coverage and unique inputs compared to baseline fuzzers.

Demonstrated effectiveness on both open-source and commercial CUDA programs.

Abstract

GPUs play an increasingly important role in modern software. However, the heterogeneous host-device execution model and expanding software stacks make GPU programs prone to memory-safety and concurrency bugs that evade static analysis. While fuzz-testing, combined with dynamic error checking tools, offers a plausible solution, it remains underutilized for GPUs. In this work, we identify three main obstacles limiting prior GPU fuzzing efforts: (1) kernel-level fuzzing leading to false positives, (2) lack of device-side coverage-guided feedback, and (3) incompatibility between coverage and sanitization tools. We present cuFuzz, the first CUDA-oriented fuzzer that makes GPU fuzzing practical by addressing these obstacles. cuFuzz uses whole program fuzzing to avoid false positives from independently fuzzing device-side kernels. It leverages NVBit to instrument device-side instructions and merges the resultant coverage with compiler-based host coverage. Finally, cuFuzz decouples sanitization from coverage collection by executing host- and device-side sanitizers in separate processes. cuFuzz uncovers 43 previously unknown bugs (19 in commercial libraries) across 14 CUDA programs, including illegal memory accesses, uninitialized reads, and data races. cuFuzz achieves significantly more discovered edges and unique inputs compared to baseline approaches, especially on closed-source targets. Moreover, we quantify the execution time overheads of the different cuFuzz components and add persistent-mode support to improve the overall fuzzing throughput. Our results demonstrate that cuFuzz is an effective and deployable addition to the GPU testing toolbox. cuFuzz is publicly available at https://github.com/NVlabs/cuFuzz/.