Characterizing Production GPU Workloads using System-wide Telemetry Data

TL;DR

This paper analyzes GPU hardware counters from a supercomputer to understand workload characteristics, revealing insights into job behavior, resource utilization, and energy consumption, which can inform system optimization and design.

Contribution

It provides a comprehensive analysis of GPU workload behavior at scale using hardware counters, introducing new metrics for workload characterization in HPC environments.

Findings

81% of jobs are memory-bound

Memory-bound jobs consume more energy

55% of large GPU jobs peak at 50% HBM capacity or less

Abstract

GPGPU-accelerated clusters and supercomputers are central to modern high-performance computing (HPC). Over the past decade, these systems continue to expand, and GPUs now expose a wide range of hardware counters that provide detailed views of performance and resource usage. Despite the potential of these counters, few studies have evaluated the insights they offer about real workloads at scale. In this work, we address this gap by analyzing previously underexplored GPU hardware counters collected via Lightweight Distributed Metric Service on Perlmutter, a leadership-class supercomputer. We quantify uneven work distribution across GPUs within a job and the steadiness of GPU activity over time, and we classify jobs as compute- or memory-bound using a roofline-based criterion. We then use these metrics to interpret job behavior in terms of practical workload characteristics to provide interpretable, job-level insights. Our findings can inform workload optimization and future HPC system design. For example, 81% of jobs are memory-bound, and memory-bound jobs tend to consume more energy than compute-bound jobs at comparable GPU-hours. Among jobs requesting 80 GB GPUs, 55% peak at 50% HBM capacity or less.