TURNIP: A "Nondeterministic" GPU Runtime with CPU RAM Offload

TL;DR

TURNIP is a GPU runtime system that intelligently manages CPU RAM offload for AI computations by dynamically scheduling operations based on real-time transfer completion, addressing nondeterminacy and improving efficiency.

Contribution

TURNIP introduces a dependency graph compilation and runtime scheduling approach to handle nondeterministic CPU-GPU memory transfers in AI workloads.

Findings

Reduces computation blocking caused by CPU offload delays.

Enables flexible operation ordering based on real-time transfer status.

Improves overall efficiency of GPU-based AI computing with CPU offload.

Abstract

An obvious way to alleviate memory difficulties in GPU-based AI computing is via CPU offload, where data are moved between GPU and CPU RAM, so inexpensive CPU RAM is used to increase the amount of storage available. While CPU offload is an obvious idea, it can greatly slow down a computation, due to the relatively slow transfer rate between CPU RAM and GPU RAM. Thus, any system for CPU offload needs to ensure that when such a transfer needs to happen, no computation is blocked waiting for the transfer to finish. One of the key challenges when using CPU offload is that memory transfers introduce nondeterminacy into the system: it is not possible to know before runtime when the transfers will finish, and hence what is the best order of operations to run to ensure there is no blocking. In this paper, we describe TURNIP, which is a system for running AI computations using CPU offload. The key innovation in TURNIP is the compilation of the AI computation into a dependency graph that gives the TURNIP runtime freedom to run operations such as GPU kernel calls in many different orders; at runtime, TURNIP chooses the best order in response to real-time events.