CurvaDion: Curvature-Adaptive Distributed Orthonormalization

TL;DR

CurvaDion adaptively reduces communication in distributed training of large language models by detecting high-curvature regions with RMMC, maintaining convergence while significantly lowering synchronization overhead.

Contribution

Introduces RMMC, a novel curvature proxy, enabling dynamic synchronization in distributed training, reducing communication by 99% without sacrificing convergence.

Findings

Achieves 99% communication reduction in large-scale training.

Maintains baseline convergence across models from 160M to 1.3B parameters.

Theoretically links RMMC to loss curvature.

Abstract

As language models scale to trillions of parameters, distributed training across many GPUs becomes essential, yet gradient synchronization over high-bandwidth, low-latency networks remains a critical bottleneck. While recent methods like Dion reduce per-step communication through low-rank updates, they synchronize at every step regardless of the optimization landscape. We observe that synchronization requirements vary dramatically throughout training: workers naturally compute similar gradients in flat regions, making frequent synchronization redundant, while high-curvature regions require coordination to prevent divergence. We introduce CurvaDion, which uses Relative Maximum Momentum Change (RMMC) to detect high-curvature regions requiring synchronization. RMMC leverages momentum dynamics which are already computed during optimization as a computationally tractable proxy for directional curvature, adding only $\mathcal{O}(d)$ operations per layer. We establish theoretical connections between RMMC and loss curvature and demonstrate that CurvaDion achieves 99\% communication reduction while matching baseline convergence across models from 160M to 1.3B parameters.