FedQueue: Queue-Aware Federated Learning for Cross-Facility HPC Training

TL;DR

FedQueue is a novel queue-aware federated learning protocol designed for HPC environments, effectively managing scheduler delays and heterogeneity to improve training efficiency and convergence.

Contribution

It introduces queue delay prediction, cutoff-based admission, and staleness-aware aggregation, with proven convergence and real-world performance improvements.

Findings

20.5% improvement over baseline algorithms in real deployment.

34% reduction in time to reach target accuracy under high queue variance.

Bounded staleness achieved with high probability despite queue prediction errors.

Abstract

Federated learning (FL) across multiple HPC facilities faces stochastic admission delays from batch schedulers that dominate wall-clock time. Synchronous FL suffers from severe stragglers, while asynchronous FL accumulates stale updates when queues spike. We propose FedQueue, a queue-aware FL protocol that incorporates scheduler delays directly into training and aggregation, which (i) predicts per-facility queue delays online to budget local work, (ii) applies cutoff-based admission that buffers late arrivals to bound staleness, and (iii) performs staleness-aware aggregation to stabilize heterogeneous local workloads. We prove the convergence for non-convex objectives at rate $\mathcal{O}(1/\sqrt{R})$ under bounded staleness, and show that the admission controls yield bounded staleness with high probability under queue-prediction error. Real-world cross-facility deployment of FedQueue shows 20.5% improvement over baseline algorithms. Controlled queue simulations demonstrate robust improvement over the baselines; in particular, about 34% reduction in time to reach a target accuracy level under high queue variance and non-IID partitions.