SIGMA: An AI-Empowered Training Stack on Early-Life Hardware

TL;DR

SIGMA is a comprehensive open-source training stack that enhances the reliability, stability, and efficiency of large-scale AI training on early-life hardware, addressing core challenges in system disruptions, numerical errors, and parallelism complexity.

Contribution

The paper introduces SIGMA, a novel training stack with the LUCIA TRAINING PLATFORM and FRAMEWORK, optimized for early-life AI accelerators, achieving high utilization, stability, and scalability in large-scale training.

Findings

94.45% effective cluster accelerator utilization

Trained 200B MoE model with 2,048 accelerators

Only one stability incident over 75 days

Abstract

An increasing variety of AI accelerators is being considered for large-scale training. However, enabling large-scale training on early-life AI accelerators faces three core challenges: frequent system disruptions and undefined failure modes that undermine reliability; numerical errors and training instabilities that threaten correctness and convergence; and the complexity of parallelism optimization combined with unpredictable local noise that degrades efficiency. To address these challenges, SIGMA is an open-source training stack designed to improve the reliability, stability, and efficiency of large-scale distributed training on early-life AI hardware. The core of this initiative is the LUCIA TRAINING PLATFORM (LTP), the system optimized for clusters with early-life AI accelerators. Since its launch in March 2025, LTP has significantly enhanced training reliability and operational productivity. Over the past five months, it has achieved an impressive 94.45% effective cluster accelerator utilization, while also substantially reducing node recycling and job-recovery times. Building on the foundation of LTP, the LUCIA TRAINING FRAMEWORK (LTF) successfully trained SIGMA-MOE, a 200B MoE model, using 2,048 AI accelerators. This effort delivered remarkable stability and efficiency outcomes, achieving 21.08% MFU, state-of-the-art downstream accuracy, and encountering only one stability incident over a 75-day period. Together, these advances establish SIGMA, which not only tackles the critical challenges of large-scale training but also establishes a new benchmark for AI infrastructure and platform innovation, offering a robust, cost-effective alternative to prevailing established accelerator stacks and significantly advancing AI capabilities and scalability. The source code of SIGMA is available at https://github.com/microsoft/LuciaTrainingPlatform.