Understanding the Landscape of Ampere GPU Memory Errors

TL;DR

This paper provides a comprehensive large-scale analysis of GPU memory errors across three supercomputers with NVIDIA A100 GPUs, revealing error patterns and reliability insights crucial for fault-tolerant HPC system design.

Contribution

It presents the first large-scale cross-supercomputer study of GPU memory errors, comparing error behaviors and offering insights for improving HPC reliability.

Findings

Error rates vary across systems but share common patterns.

Error characteristics differ between systems and over time.

Insights inform better fault tolerance and checkpointing strategies.

Abstract

Graphics Processing Units (GPUs) have become a de facto solution for accelerating high-performance computing (HPC) applications. Understanding their memory error behavior is an essential step toward achieving efficient and reliable HPC systems. In this work, we present a large-scale cross-supercomputer study to characterize GPU memory reliability, covering three supercomputers - Delta, Polaris, and Perlmutter - all equipped with NVIDIA A100 GPUs. We examine error logs spanning 67.77 million GPU device-hours across 10,693 GPUs. We compare error rates and mean-time-between-errors (MTBE) and highlight both shared and distinct error characteristics among these three systems. Based on these observations and analyses, we discuss the implications and lessons learned, focusing on the reliable operation of supercomputers, the choice of checkpointing interval, and the comparison of reliability characteristics with those of previous-generation GPUs. Our characterization study provides valuable insights into fault-tolerant HPC system design and operation, enabling more efficient execution of HPC applications.