Towards Efficient and Practical GPU Multitasking in the Era of LLM

TL;DR

This paper advocates for a GPU multitasking paradigm inspired by CPU OS, emphasizing the need for a resource management layer to handle diverse AI workloads efficiently.

Contribution

It identifies the limitations of current GPU singletasking and proposes a new resource management framework to enable practical GPU multitasking for modern AI workloads.

Findings

Current GPU singletasking is inefficient for AI workloads.

A resource management layer can enable effective GPU multitasking.

The paper outlines challenges and potential solutions for GPU multitasking.

Abstract

GPU singletasking is becoming increasingly inefficient and unsustainable as hardware capabilities grow and workloads diversify. We are now at an inflection point where GPUs must embrace multitasking, much like CPUs did decades ago, to meet the demands of modern AI workloads. In this work, we highlight the key requirements for GPU multitasking, examine prior efforts, and discuss why they fall short. To advance toward efficient and practical GPU multitasking, we envision a resource management layer, analogous to a CPU operating system, to handle various aspects of GPU resource management and sharing. We outline the challenges and potential solutions, and hope this paper inspires broader community efforts to build the next-generation GPU compute paradigm grounded in multitasking.