Hawkeye: Reproducing GPU-Level Non-Determinism

TL;DR

Hawkeye is a system that accurately reproduces GPU-level matrix multiplication operations on a CPU, enabling trustworthy third-party auditing of machine learning workflows without precision loss.

Contribution

Hawkeye introduces a systematic testing framework to reproduce GPU matrix operations on CPU with perfect accuracy across multiple architectures and precisions.

Findings

Hawkeye achieves perfect reproduction of GPU matrix multiplication on CPU.

The framework works across NVIDIA GPU architectures and precision types.

Hawkeye enables reliable third-party auditing of ML training and inference.

Abstract

We present Hawkeye, a system for analyzing and reproducing GPU-level arithmetic operations. Using our framework, anyone can re-execute on a CPU the exact matrix multiplication operations underlying a machine learning model training or inference workflow that was executed on an NVIDIA GPU, without any precision loss. This is in stark contrast to prior approaches to verifiable machine learning, which either introduce significant computation overhead to the original model owner, or suffer from non-robustness and quality degradation. The main technical contribution of Hawkeye is a systematic sequence of carefully crafted tests that study rounding direction, subnormal number handling, and order of (non-associative) accumulation during matrix multiplication on NVIDIA's Tensor Cores. We test and evaluate our framework on multiple NVIDIA GPU architectures ( Ampere, Hopper, and Lovelace) and precision types (FP16, BFP16, FP8). In all test cases, Hawkeye enables perfect reproduction of matrix multiplication on a CPU, paving the way for efficient and trustworthy third-party auditing of ML model training and inference. We provide source code for Hawkeye at https://github.com/badasherez/gpu-simulator.