LUMION: Fast Fault Recovery for ML Jobs Using Programmable Optical Fabrics

TL;DR

LUMION introduces a reconfigurable optical fabric that dynamically replaces failed accelerators in ML datacenter racks, significantly improving fault recovery speed and throughput without full job migrations.

Contribution

LUMION presents a novel optical fabric enabling rapid, in-rack replacement of failed accelerators, reducing resource waste and maintaining high performance during faults.

Findings

LUMION swaps failed GPUs within ~1 second.

Achieves nearly 2X improvement in fine-tuning throughput.

Provides higher inter-GPU bandwidth compared to traditional racks.

Abstract

When accelerators fail in modern ML datacenters, operators migrate the affected ML training or inference jobs to entirely new racks. This approach, while preserving network performance, is highly inefficient, requiring datacenters to reserve full racks of idle accelerators for fault tolerance. In this paper, we address this resource inefficiency by introducing LUMION, a novel reconfigurable optical fabric for connecting accelerators within a datacenter rack. Instead of migrating entire ML jobs, LUMION dynamically integrates spare accelerators into ongoing workloads as failures occur, thereby maintaining consistent performance without costly migrations. We show the benefits of LUMION by building an end-to-end hardware prototype. Our experiments fine-tune Llama 3.2 and show that LUMION swaps a failed GPU with a healthy one and restarts the ML job within ~ 1 second of the failure. LUMION achieves higher inter-GPU bandwidth compared to traditional electrical racks after replacing failed accelerators with spare ones, leading to nearly 2X improvement in fine-tuning throughput.