An Analysis of Collocation on GPUs for Deep Learning Training

TL;DR

This paper evaluates GPU collocation techniques, including Multi-Instance GPU (MIG), for deep learning training, showing that collocation can improve throughput but depends on workload characteristics and resource partitioning.

Contribution

It provides a comparative analysis of MIG, traditional process collocation, and MPS for GPU utilization in deep learning training, highlighting their advantages and limitations.

Findings

Collocating multiple models can increase training throughput up to four times.

MIG offers interference-free partitioning but may underutilize resources with dynamic workloads.

MPS is identified as the most flexible and effective method for GPU collocation.

Abstract

Deep learning training is an expensive process that extensively uses GPUs, but not all model training saturates modern powerful GPUs. Multi-Instance GPU (MIG) is a new technology introduced by NVIDIA that can partition a GPU to better-fit workloads that do not require all the memory and compute resources of a full GPU. In this paper, we examine the performance of a MIG-enabled A100 GPU under deep learning workloads containing various sizes and combinations of models. We contrast the benefits of MIG to older workload collocation methods on GPUs: na\"ively submitting multiple processes on the same GPU and utilizing Multi-Process Service (MPS). Our results demonstrate that collocating multiple model training runs may yield significant benefits. In certain cases, it can lead up to four times training throughput despite increased epoch time. On the other hand, the aggregate memory footprint and compute needs of the models trained in parallel must fit the available memory and compute resources of the GPU. MIG can be beneficial thanks to its interference-free partitioning, especially when the sizes of the models align with the MIG partitioning options. MIG's rigid partitioning, however, may create sub-optimal GPU utilization for more dynamic mixed workloads. In general, we recommend MPS as the best performing and most flexible form of collocation for model training for a single user submitting training jobs.