EF21 with Bells & Whistles: Six Algorithmic Extensions of Modern Error Feedback
Ilyas Fatkhullin, Igor Sokolov, Eduard Gorbunov, Zhize Li, Peter Richt\'arik

TL;DR
This paper introduces six practical extensions of the EF21 error feedback mechanism for distributed optimization, supported by strong convergence theory, improving upon previous methods in both theory and practice.
Contribution
Six novel algorithmic extensions of EF21 are proposed, with comprehensive convergence analysis, enhancing its applicability and performance in distributed optimization.
Findings
All extensions are supported by strong convergence guarantees.
Theoretical improvements over existing error feedback methods.
Enhanced practical performance demonstrated in experiments.
Abstract
First proposed by Seide (2014) as a heuristic, error feedback (EF) is a very popular mechanism for enforcing convergence of distributed gradient-based optimization methods enhanced with communication compression strategies based on the application of contractive compression operators. However, existing theory of EF relies on very strong assumptions (e.g., bounded gradients), and provides pessimistic convergence rates (e.g., while the best known rate for EF in the smooth nonconvex regime, and when full gradients are compressed, is , the rate of gradient descent in the same regime is ). Recently, Richt\'arik et al. (2021) proposed a new error feedback mechanism, EF21, based on the construction of a Markov compressor induced by a contractive compressor. EF21 removes the aforementioned theoretical deficiencies of EF and at the same time works better in practice. In…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNumerical Methods and Algorithms
