Flare: Anomaly Diagnostics for Divergent LLM Training in GPU Clusters of Thousand-Plus Scale

TL;DR

Flare is a comprehensive diagnostic framework for large-scale distributed LLM training that automatically detects anomalies and performance issues across the entire training stack, improving reliability on thousands of GPUs.

Contribution

It introduces a scalable, full-stack diagnostic system with automatic anomaly detection tailored for large GPU clusters training LLMs, addressing gaps in existing tools.

Findings

Effective anomaly diagnosis across 6,000 GPUs

Continuous operation for over eight months

Significant improvements in pinpointing training deficiencies

Abstract

The rapid proliferation of large language models has driven the need for efficient GPU training clusters. However, it is challenging due to the frequent occurrence of training anomalies. Since existing diagnostic tools are narrowly tailored to specific issues, there are gaps in their ability to address anomalies spanning the entire training stack. In response, we introduce Flare, a diagnostic framework designed for distributed LLM training at scale. Flare first integrates a lightweight tracing daemon for full-stack and backend-extensible tracing. Additionally, it features a diagnostic engine that automatically diagnoses anomalies, with a focus on performance regressions. The deployment of Flare across 6,000 GPUs has demonstrated significant improvements in pinpointing deficiencies in real-world scenarios, with continuous operation for over eight months.