NetSenseML: Network-Adaptive Compression for Efficient Distributed Machine Learning

TL;DR

NetSenseML is a dynamic framework that adaptively adjusts gradient compression techniques based on real-time network conditions to optimize distributed machine learning training efficiency without sacrificing accuracy.

Contribution

It introduces a novel network adaptive approach that balances compression and accuracy by monitoring network conditions during training.

Findings

Improves training throughput by up to 9.84x in bandwidth-constrained environments.

Effectively balances network load and model accuracy during distributed training.

Demonstrates significant performance gains over existing compression methods.

Abstract

Training large-scale distributed machine learning models imposes considerable demands on network infrastructure, often resulting in sudden traffic spikes that lead to congestion, increased latency, and reduced throughput, which would ultimately affect convergence times and overall training performance. While gradient compression techniques are commonly employed to alleviate network load, they frequently compromise model accuracy due to the loss of gradient information. This paper introduces NetSenseML, a novel network adaptive distributed deep learning framework that dynamically adjusts quantization, pruning, and compression strategies in response to real-time network conditions. By actively monitoring network conditions, NetSenseML applies gradient compression only when network congestion negatively impacts convergence speed, thus effectively balancing data payload reduction and model accuracy preservation. Our approach ensures efficient resource usage by adapting reduction techniques based on current network conditions, leading to shorter convergence times and improved training efficiency. We present the design of the NetSenseML adaptive data reduction function and experimental evaluations show that NetSenseML can improve training throughput by a factor of 1.55 to 9.84 times compared to state-of-the-art compression-enabled systems for representative DDL training jobs in bandwidth-constrained conditions.