A Practical Two-Stage Framework for GPU Resource and Power Prediction in Heterogeneous HPC Systems

TL;DR

This paper presents a two-stage framework for predicting GPU resource utilization and power consumption in HPC systems, leveraging workload logs and GPU profiling data for improved scheduling and power management.

Contribution

It introduces a novel two-stage prediction framework that combines workload logs and GPU metrics to accurately forecast GPU utilization and power in heterogeneous HPC environments.

Findings

GPU utilization prediction accuracy up to 97% using Slurm logs

Power prediction accuracy up to 92% with GPU metrics

GPU profiling metrics effectively capture application characteristics

Abstract

Efficient utilization of GPU resources and power has become critical with the growing demand for GPUs in high-performance computing (HPC). In this paper, we analyze GPU utilization and GPU memory utilization, as well as the power consumption of the Vienna ab initio Simulation Package (VASP), using the Slurm workload manager historical logs and GPU performance metrics collected by NVIDIA's Data Center GPU Manager (DCGM). VASP is a widely used materials science application on Perlmutter at NERSC, an HPE Cray EX system based on NVIDIA A100 GPUs. Using our insights from the resource utilization analysis of VASP applications, we propose a resource prediction framework to predict the average GPU power, maximum GPU utilization, and maximum GPU memory utilization values of heterogeneous HPC system applications to enable more efficient scheduling decisions and power-aware system operation. Our prediction framework consists of two stages: 1) using only the Slurm accounting logs as training data and 2) augmenting the training data with historical GPU profiling metrics collected with DCGM. The maximum GPU utilization predictions using only the Slurm submission features achieve up to 97% accuracy. Furthermore, features engineered from GPU-compute and memory activity metrics exhibit good correlations with average power utilization, and our runtime power usage prediction experiments result in up to 92% prediction accuracy. These findings demonstrate the effectiveness of DCGM metrics in capturing application characteristics and highlight their potential for developing predictive models to support dynamic power management in HPC systems.