Understanding the Landscape of Ampere GPU Memory Errors
Zhu Zhu, Yu Sun, Dhatri Parakal, Bo Fang, Steven Farrell, Gregory H. Bauer, Brett Bode, Ian T. Foster, Michael E. Papka, William Gropp, Zhao Zhang, Lishan Yang

TL;DR
This paper provides a comprehensive large-scale analysis of GPU memory errors across three supercomputers with NVIDIA A100 GPUs, revealing error patterns and reliability insights crucial for fault-tolerant HPC system design.
Contribution
It presents the first large-scale cross-supercomputer study of GPU memory errors, comparing error behaviors and offering insights for improving HPC reliability.
Findings
Error rates vary across systems but share common patterns.
Error characteristics differ between systems and over time.
Insights inform better fault tolerance and checkpointing strategies.
Abstract
Graphics Processing Units (GPUs) have become a de facto solution for accelerating high-performance computing (HPC) applications. Understanding their memory error behavior is an essential step toward achieving efficient and reliable HPC systems. In this work, we present a large-scale cross-supercomputer study to characterize GPU memory reliability, covering three supercomputers - Delta, Polaris, and Perlmutter - all equipped with NVIDIA A100 GPUs. We examine error logs spanning 67.77 million GPU device-hours across 10,693 GPUs. We compare error rates and mean-time-between-errors (MTBE) and highlight both shared and distinct error characteristics among these three systems. Based on these observations and analyses, we discuss the implications and lessons learned, focusing on the reliable operation of supercomputers, the choice of checkpointing interval, and the comparison of reliability…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
