Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods

TL;DR

This paper proposes adaptive batch size strategies for distributed local gradient methods to reduce communication costs and improve training efficiency and generalization, supported by theoretical guarantees and empirical results.

Contribution

It introduces novel adaptive batch size strategies for local gradient methods with convergence guarantees and demonstrates their effectiveness through experiments.

Findings

Reduced communication overhead in distributed training.

Improved training efficiency and model generalization.

Effective adaptive batch size strategies validated by experiments.

Abstract

Modern deep neural networks often require distributed training with many workers due to their large size. As the number of workers increases, communication overheads become the main bottleneck in data-parallel minibatch stochastic gradient methods with per-iteration gradient synchronization. Local gradient methods like Local SGD reduce communication by only synchronizing model parameters and/or gradients after several local steps. Despite an understanding of their convergence and the importance of batch sizes for training efficiency and generalization, optimal batch sizes for local gradient methods are difficult to determine. We introduce adaptive batch size strategies for local gradient methods that increase batch sizes adaptively to reduce minibatch gradient variance. We provide convergence guarantees under homogeneous data conditions and support our claims with image classification and language modeling experiments, demonstrating the effectiveness of our strategies for both training efficiency and generalization.