Exploring Silent Data Corruption as a Reliability Challenge in LLM Training

TL;DR

This paper investigates how silent data corruption during GPU-based training of large language models can cause significant training issues, and proposes a detection method to mitigate these effects.

Contribution

It provides a controlled fault injection study of SDC in LLM training and introduces a lightweight detection approach to improve training robustness.

Findings

Faults in GPU matrix operations can cause loss spikes and divergence.

Recomputing recent training steps can mitigate SDC impacts.

Certain bit positions and kernel functions are more sensitive to faults.

Abstract

As Large Language Models (LLMs) scale in size and complexity, the consequences of failures during training become increasingly severe. A major challenge arises from Silent Data Corruption (SDC): hardware-induced faults that bypass system-level detection mechanisms. SDC may behave like benign numerical noise, but can also cause harmful gradient corruption that leads to loss spikes, divergence, or stalled progress. This work provides a controlled study of how intermittent SDC affects LLM pretraining. Using targeted fault injection at the level of GPU matrix-multiply instructions, we characterize the sensitivity of different bit positions, kernel functions, and execution stages. Our analysis shows that locally originating faults can produce impactful corruption, including NaN propagation, short-lived spikes in loss, gradient norm, and attention logits, as well as persistent parameter divergence. Building on the observed corruption signatures, we propose a lightweight detection method that identifies potentially harmful parameter updates. Experiments on LLaMA models with 60M, 350M, and 1.3B parameters demonstrate that recomputing the most recent training step upon detection can effectively mitigate the impact of these events.