Improved Convergence in Parameter-Agnostic Error Feedback through Momentum

TL;DR

This paper introduces parameter-agnostic normalized error feedback algorithms with momentum for distributed training, achieving near-optimal convergence rates without problem-specific tuning, validated by theoretical analysis and experiments.

Contribution

It proposes new normalized EF algorithms with momentum that do not require prior problem parameter knowledge, improving practical applicability in distributed neural network training.

Findings

Normalized EF21 achieves near O(1/T^{1/4}) convergence with heavy-ball momentum.

Algorithms attain convergence rates close to tuned methods without parameter tuning.

Empirical results validate theoretical convergence bounds.

Abstract

Communication compression is essential for scalable distributed training of modern machine learning models, but it often degrades convergence due to the noise it introduces. Error Feedback (EF) mechanisms are widely adopted to mitigate this issue of distributed compression algorithms. Despite their popularity and training efficiency, existing distributed EF algorithms often require prior knowledge of problem parameters (e.g., smoothness constants) to fine-tune stepsizes. This limits their practical applicability especially in large-scale neural network training. In this paper, we study normalized error feedback algorithms that combine EF with normalized updates, various momentum variants, and parameter-agnostic, time-varying stepsizes, thus eliminating the need for problem-dependent tuning. We analyze the convergence of these algorithms for minimizing smooth functions, and establish parameter-agnostic complexity bounds that are close to the best-known bounds with carefully-tuned problem-dependent stepsizes. Specifically, we show that normalized EF21 achieve the convergence rate of near ${O}(1/T^{1/4})$ for Polyak's heavy-ball momentum, ${O}(1/T^{2/7})$ for Iterative Gradient Transport (IGT), and ${O}(1/T^{1/3})$ for STORM and Hessian-corrected momentum. Our results hold with decreasing stepsizes and small mini-batches. Finally, our empirical experiments confirm our theoretical insights.