Automatic Stability and Recovery for Neural Network Training

TL;DR

This paper presents a supervisory framework for neural network training that detects and recovers from instability during optimization, ensuring safer training without altering existing optimizers.

Contribution

It introduces a novel runtime stability framework using secondary measurements for automatic detection and recovery, with theoretical safety guarantees and minimal overhead.

Findings

Framework effectively detects destabilizing updates in real-time.

Automatic recovery maintains training stability without optimizer modifications.

The approach is compatible with memory-constrained environments.

Abstract

Training modern neural networks is increasingly fragile, with rare but severe destabilizing updates often causing irreversible divergence or silent performance degradation. Existing optimization methods primarily rely on preventive mechanisms embedded within the optimizer, offering limited ability to detect and recover from instability once it occurs. We introduce a supervisory runtime stability framework that treats optimization as a controlled stochastic process. By isolating an innovation signal derived from secondary measurements, such as validation probes, the framework enables automatic detection and recovery from destabilizing updates without modifying the underlying optimizer. We provide theoretical runtime safety guarantees that formalize bounded degradation and recovery. Our implementation incurs minimal overhead and is compatible with memory-constrained training settings.