Distributed Training under Packet Loss

TL;DR

This paper introduces a distributed training framework that maintains model accuracy and convergence guarantees over unreliable network connections with packet loss, enabling scalable training across commodity networks.

Contribution

A novel end-to-end distributed training method that ensures unbiased gradient aggregation and bounded parameter drift under packet loss without changing model code or optimizers.

Findings

Tolerates 10% packet loss with less than 1% perplexity increase on LLAMA2 7B.

Provides theoretical guarantees for unbiased gradients and bounded parameter divergence.

Demonstrates robustness and scalability on large models and multi-GPU setups.

Abstract

State-of-the-art language and vision models are routinely trained across thousands of GPUs, often spanning multiple data-centers, yet today's distributed frameworks still assume reliable connections (e.g., InfiniBand or RoCE). The resulting acknowledgment traffic and retransmissions inflate tail latencies and limit scalability. Leveraging unreliable connections will reduce latency but may sacrifice model accuracy and convergence once packets are dropped. A principled, end-to-end solution that preserves accuracy and convergence guarantees under genuine packet loss has previously been missing. We address this critical gap by introducing a novel distributed training framework capable of operating over unreliable connections, offering unbiased gradient aggregation and bounded parameter drift without modifying model code or optimizers. The key insight is a two-stage defense against missing messages: (i) Unbiased gradient aggregation: each worker reconstructs a consistent gradient estimate from whatever packets arrive, guaranteeing expectation-level correctness; and (ii) Bounded-drift parameter broadcasts: we prove the inter-worker model discrepancy remains O(1) even after arbitrarily many iterations, preventing the unbounded divergence typical of asynchronous setups. Analytical bounds are matched by experiments on the LLAMA2 7B model with 64 GPUs: tolerating 10% random packet loss yields at most 0.8% perplexity change. This work bridges the gap between communication-efficient datacenter protocols and the accuracy and generalization guarantees demanded by modern large-model training, enabling robust, high-throughput learning on commodity or wide-area networks.