Sign Bit is Enough: A Learning Synchronization Framework for Multi-hop All-reduce with Ultimate Compression

TL;DR

This paper introduces Marsit, a sign-bit compression-based synchronization framework for multi-hop all-reduce that maintains convergence rates and reduces training time by up to 35% without sacrificing accuracy.

Contribution

Marsit is the first framework to effectively prevent cascading compression in multi-hop all-reduce, ensuring unbiased sign aggregation and convergence preservation.

Findings

Reduces training time by up to 35%.

Maintains the same convergence rate as non-compression methods.

Effectively mitigates cascading compression effects.

Abstract

Traditional one-bit compressed stochastic gradient descent can not be directly employed in multi-hop all-reduce, a widely adopted distributed training paradigm in network-intensive high-performance computing systems such as public clouds. According to our theoretical findings, due to the cascading compression, the training process has considerable deterioration on the convergence performance. To overcome this limitation, we implement a sign-bit compression-based learning synchronization framework, Marsit. It prevents cascading compression via an elaborate bit-wise operation for unbiased sign aggregation and its specific global compensation mechanism for mitigating compression deviation. The proposed framework retains the same theoretical convergence rate as non-compression mechanisms. Experimental results demonstrate that Marsit reduces up to 35% training time while preserving the same accuracy as training without compression.