LLM-PRISM: Characterizing Silent Data Corruption from Permanent GPU Faults in LLM Training

TL;DR

This paper introduces LLM-PRISM, a methodology combining RTL GPU fault simulation and stochastic injection to analyze the resilience of large language model training against silent data corruption caused by hardware faults.

Contribution

It provides the first hardware-grounded characterization of LLM pre-training resilience to silent data corruption from GPU faults, analyzing fault effects across multiple numeric formats.

Findings

LLMs resist low-frequency faults but with non-uniform impact.

Critical datapaths and certain formats can cause catastrophic divergence.

Resilience varies significantly with fault type, rate, and numeric format.

Abstract

Large-scale LLM training is increasingly susceptible to hardware defects stemming from manufacturing escapes and silicon aging. These defects manifest as Silent Data Corruption (SDC) that perturb gradients and parameters throughout the training process. We present LLM-PRISM, a methodology to characterize LLM pre-training resilience to hardware faults. LLM-PRISM couples RTL-level GPU fault simulation with a stochastic injection engine embedded in Megatron-LM. Through 7,664 training runs across FP16, BF16, and FP8 regimes, we analyze how fault type, rate, and numeric format govern resilience. We find that while LLMs resist low-frequency faults, impact is highly non-uniform; critical datapaths and specific precision formats can induce catastrophic divergence even at moderate fault rates. This study provides the first hardware-grounded, pre-training characterization of SDC resilience.