DynaTrain: Fast Online Parallelism Switching for Elastic LLM Training

TL;DR

DynaTrain is a system enabling rapid, sub-second online reconfiguration of parallelism in large language model training, adapting to resource changes efficiently.

Contribution

It introduces the Virtual Parameter Space abstraction and a novel scheduling method for fast, deadlock-free parallelism switching during LLM training.

Findings

Reconfigures a 70B dense model in under 2 seconds.

Reconfigures a 235B MoE model in 4.36 seconds.

Outperforms existing systems by up to three orders of magnitude.

Abstract

Modern large language model (LLM) training is inherently dynamic: resource fluctuations, RLHF phase shifts, and cluster elasticity continually reshape the optimal parallelism layout, posing a significant challenge to existing training frameworks built around a static execution model. We present DynaTrain, a distributed training system for sub-second, online reconfiguration across arbitrary multi-dimensional parallelism. At its core, we propose a Virtual Parameter Space (VPS) abstraction that unifies all distributed training states under one logical coordinate space, turning any parallelism configuration into a deterministic mapping and collapsing complex transition into manageable geometric intersections. On top of VPS, a state routing-and-transition layer executes rank-local transfers under a memory-aware, deadlock-free schedule, and an Elastic Device Manager overlaps new-world construction with ongoing training to mask topology-change cost. On dense and MoE models up to 235B parameters, DynaTrain reconfigures a 70B dense model in under 2s and a 235B MoE model in 4.36s, outperforming state-of-the-art checkpoint-based and elastic systems by up to three orders of magnitude while preserving correctness.