HetGPU: The pursuit of making binary compatibility towards GPUs

TL;DR

HetGPU is a system that enables a single GPU binary to run across multiple vendors' hardware by using an architecture-agnostic IR, dynamic translation, and abstraction layers, facilitating vendor-agnostic GPU computing.

Contribution

The paper introduces hetGPU, a novel system that allows binary compatibility across diverse GPU architectures through IR-based compilation and runtime translation.

Findings

Unmodified GPU binaries can be migrated across different hardware with minimal overhead.

HetGPU effectively abstracts diverse execution models and instruction sets.

Preliminary results show promising vendor-agnostic GPU execution capabilities.

Abstract

Heterogeneous GPU infrastructures present a binary compatibility challenge: code compiled for one vendor's GPU will not run on another due to divergent instruction sets, execution models, and driver stacks . We propose hetGPU, a new system comprising a compiler, runtime, and abstraction layer that together enable a single GPU binary to execute on NVIDIA, AMD, Intel, and Tenstorrent hardware. The hetGPU compiler emits an architecture-agnostic GPU intermediate representation (IR) and inserts metadata for managing execution state. The hetGPU runtime then dynamically translates this IR to the target GPU's native code and provides a uniform abstraction of threads, memory, and synchronization. Our design tackles key challenges: differing SIMT vs. MIMD execution (warps on NVIDIA/AMD vs. many-core RISC-V on Tenstorrent), varied instruction sets, scheduling and memory model discrepancies, and the need for state serialization for live migration. We detail the hetGPU architecture, including the IR transformation pipeline, a state capture/reload mechanism for live GPU migration, and an abstraction layer that bridges warp-centric and core-centric designs. Preliminary evaluation demonstrates that unmodified GPU binaries compiled with hetGPU can be migrated across disparate GPUs with minimal overhead, opening the door to vendor-agnostic GPU computing.