GPUReplay: A 50-KB GPU Stack for Client ML

TL;DR

GPUReplay is a compact, portable GPU stack that records and replays GPU executions for mobile and embedded devices, simplifying deployment and enhancing security for machine learning workloads.

Contribution

It introduces a tiny, robust GPU replay system that replaces the original GPU stack, enabling secure, fast, and portable GPU execution on resource-constrained devices.

Findings

Replays long GPU executions without divergence

Speeds up startup by up to two orders of magnitude

Works across various hardware, APIs, and neural network models

Abstract

GPUReplay (GR) is a novel way for deploying GPU-accelerated computation on mobile and embedded devices. It addresses high complexity of a modern GPU stack for deployment ease and security. The idea is to record GPU executions on the full GPU stack ahead of time and replay the executions on new input at run time. We address key challenges towards making GR feasible, sound, and practical to use. The resultant replayer is a drop-in replacement of the original GPU stack. It is tiny (50 KB of executable), robust (replaying long executions without divergence), portable (running in a commodity OS, in TEE, and baremetal), and quick to launch (speeding up startup by up to two orders of magnitude). We show that GPUReplay works with a variety of integrated GPU hardware, GPU APIs, ML frameworks, and 33 neural network (NN) implementations for inference or training. The code is available at https://github.com/bakhi/GPUReplay.