Symphony: Taming Step Misalignments in the Network for Ring-based Collective Operations

TL;DR

Symphony is an in-network approach that detects and mitigates step misalignments in ring-based collective operations, significantly improving communication efficiency in distributed AI training.

Contribution

It introduces a lightweight in-network mechanism to track progress and use congestion signals for selective throttling, addressing network jitter issues.

Findings

Up to 54% reduction in collective communication time.

Effective mitigation of step misalignments in simulations.

Prototype validation on a programmable switch demonstrates practicality.

Abstract

Ring-based collective operations are widely used in distributed AI training due to their efficient bandwidth utilization. While ring communication excels at pipelining, its performance is heavily dependent on having synchronized step-wise progression. This presents a mismatch to the underlying network conditions in practice: collective operations are vulnerable to network jitter and congestion, leading to step misalignment and increased collective completion time. To that end, we propose Symphony, an in-network solution that detects pipeline step misalignment and mitigates its impact. Symphony introduces (1) a lightweight mechanism to track per-job pipeline progress and (2) a novel use of congestion signals to selectively throttle outpacing flows, allowing lagging flows to catch up without global coordination. Through simulations using Astra-Sim, we show that Symphony effectively mitigates step misalignments in ring-based collectives, resulting in up to 54% improvement in job/collective communication time. Finally, we prototype and validate Symphony on an Intel Tofino2 programmable switch to demonstrate its practicality.