Optimal Workload Placement on Multi-Instance GPUs

TL;DR

This paper presents methods to optimize GPU workload placement, significantly reducing GPU usage and wastage for large language model inference workloads on multi-instance GPUs.

Contribution

It introduces an optimization and a heuristic approach for efficient GPU workload placement, addressing practical use cases and outperforming baseline heuristics.

Findings

Up to 2.85x reduction in GPU count

Up to 70% decrease in GPU wastage

Effective for large language model inference workloads

Abstract

There is an urgent and pressing need to optimize usage of Graphical Processing Units (GPUs), which have arguably become one of the most expensive and sought after IT resources. To help with this goal, several of the current generation of GPUs support a partitioning feature, called Multi-Instance GPU (MIG) to allow multiple workloads to share a GPU, albeit with some constraints. In this paper we investigate how to optimize the placement of Large Language Model (LLM)-based AI Inferencing workloads on GPUs. We first identify and present several use cases that are encountered in practice that require workloads to be efficiently placed or migrated to other GPUs to make room for incoming workloads. The overarching goal is to use as few GPUs as possible and to further minimize memory and compute wastage on GPUs that are utilized. We have developed two approaches to address this problem: an optimization method and a heuristic method. We benchmark these with two workload scheduling heuristics for multiple use cases. Our results show up to 2.85x improvement in the number of GPUs used and up to 70% reduction in GPU wastage over baseline heuristics. We plan to enable the SRE community to leverage our proposed method in production environments.