On the Partitioning of GPU Power among Multi-Instances

TL;DR

This paper develops software-based methods, including machine learning models, to accurately estimate power consumption of individual GPU partitions in multi-instance GPUs, addressing a key challenge for energy-efficient cloud computing.

Contribution

It introduces online, workload-specific power models that improve accuracy of power attribution among GPU partitions, overcoming limitations of generic models.

Findings

ML-based models outperform generic models in accuracy

Online models adapt to diverse workloads and concurrent usage

Demonstrated on NVIDIA A100 GPUs with ML and LLM workloads

Abstract

Efficient power management in cloud data centers is essential for reducing costs, enhancing performance, and minimizing environmental impact. GPUs, critical for tasks like machine learning (ML) and GenAI, are major contributors to power consumption. NVIDIA's Multi-Instance GPU (MIG) technology improves GPU utilization by enabling isolated partitions with per-partition resource tracking, facilitating GPU sharing by multiple tenants. However, accurately apportioning GPU power consumption among MIG instances remains challenging due to a lack of hardware support. This paper addresses this challenge by developing software methods to estimate power usage per MIG partition. We analyze NVIDIA GPU utilization metrics and find that light-weight methods with good accuracy can be difficult to construct. We hence explore the use of ML-based power models to enable accurate, partition-level power estimation. Our findings reveal that a single generic offline power model or modeling method is not applicable across diverse workloads, especially with concurrent MIG usage, and that online models constructed using partition-level utilization metrics of workloads under execution can significantly improve accuracy. Using NVIDIA A100 GPUs, we demonstrate this approach for accurate partition-level power estimation for workloads including matrix multiplication and Large Language Model inference, contributing to transparent and fair carbon reporting.