GPU Under Pressure: Estimating Application's Stress via Telemetry and Performance Counters

TL;DR

This paper proposes a method to estimate GPU stress caused by applications using telemetry data and performance counters, aiming to predict reliability issues and aging effects in high-performance computing environments.

Contribution

It introduces a novel approach combining telemetry and performance counters to accurately assess GPU stress during application execution.

Findings

Stress correlates with telemetry and performance counter data.

Combining counters on throughput, instructions, and stalls improves stress estimation.

Method enables proactive reliability and aging management in GPU workloads.

Abstract

Graphics Processing Units (GPUs) are specialized accelerators in data centers and high-performance computing (HPC) systems, enabling the fast execution of compute-intensive applications, such as Convolutional Neural Networks (CNNs). However, sustained workloads can impose significant stress on GPU components, raising reliability concerns due to potential faults that corrupt the intermediate application computations, leading to incorrect results. Estimating the stress induced by an application is thus crucial to predict reliability (with\,special\,emphasis\,on\,aging\,effects). In this work, we combine online telemetry parameters and hardware performance counters to assess GPU stress induced by different applications. The experimental results indicate the stress induced by a parallel workload can be estimated by combining telemetry data and Performance Counters that reveal the efficiency in the resource usage of the target workload. For this purpose the selected performance counters focus on measuring the i) throughput, ii) amount of issued instructions and iii) stall events.