Analyzing Resource Utilization in an HPC System: A Case Study of NERSC Perlmutter

TL;DR

This study analyzes resource utilization in NERSC's Perlmutter HPC system, revealing significant underutilization of CPU and GPU resources, and highlights the need for more fine-grained resource allocation policies.

Contribution

It provides an early, detailed analysis of resource usage patterns in Perlmutter, emphasizing the potential for improved resource management strategies.

Findings

CPUs are often underutilized, especially in GPU jobs

Approximately 64% of jobs used 50% or less of memory

Half of GPU jobs used up to 25% of GPU memory

Abstract

Resource demands of HPC applications vary significantly. However, it is common for HPC systems to primarily assign resources on a per-node basis to prevent interference from co-located workloads. This gap between the coarse-grained resource allocation and the varying resource demands can lead to HPC resources being not fully utilized. In this study, we analyze the resource usage and application behavior of NERSC's Perlmutter, a state-of-the-art open-science HPC system with both CPU-only and GPU-accelerated nodes. Our one-month usage analysis reveals that CPUs are commonly not fully utilized, especially for GPU-enabled jobs. Also, around 64% of both CPU and GPU-enabled jobs used 50% or less of the available host memory capacity. Additionally, about 50% of GPU-enabled jobs used up to 25% of the GPU memory, and the memory capacity was not fully utilized in some ways for all jobs. While our study comes early in Perlmutter's lifetime thus policies and application workload may change, it provides valuable insights on performance characterization, application behavior, and motivates systems with more fine-grain resource allocation.