gpu_ext: Extensible OS Policies for GPUs via eBPF

TL;DR

gpu_ext introduces an eBPF-based framework that enables programmable, safe, and extensible OS policies for GPUs, improving performance and latency across diverse workloads without requiring application modifications.

Contribution

It presents a novel eBPF-based runtime for GPUs that allows safe, flexible policy enforcement directly within GPU drivers and kernels, addressing limitations of existing approaches.

Findings

Up to 4.8x throughput improvement

Up to 2x tail latency reduction

Low overhead with no application modifications

Abstract

Performance in modern GPU-centric systems increasingly depends on resource management policies, including memory placement, scheduling, and observability. However, uniform policies typically yield suboptimal performance across diverse workloads. Existing approaches present a tradeoff: user-space runtimes provide programmability and flexibility but lack cross-tenant visibility and fine-grained control of hardware resources; meanwhile, modifications to the OS kernel introduce significant complexity and safety risks. To address this, we argue that the GPU driver and device layer should provide an extensible OS interface for policy enforcement. While the emerging eBPF technology shows potential, directly applying existing host-side eBPF is insufficient because they lack visibility and control into critical device-side events, and directly embedding policy code into GPU kernels could compromise safety and efficiency. We propose gpu_ext, an eBPF-based runtime that treats the GPU driver and device as a programmable OS subsystem. gpu_ext extends GPU drivers by exposing safe programmable hooks and introduces a device-side eBPF runtime capable of executing verified policy logic within GPU kernels, enabling coherent and transparent policies. Evaluation across realistic workloads including inference, training, and vector search demonstrates that gpu_ext improves throughput by up to 4.8x and reduces tail latency by up to 2x, incurring low overhead, without modifying or restarting applications