Faster Adaptive Decentralized Learning Algorithms

TL;DR

This paper introduces faster adaptive decentralized algorithms for nonconvex stochastic and finite-sum optimization, achieving near-optimal sample complexities and demonstrating improved efficiency over existing methods.

Contribution

Proposes novel adaptive decentralized algorithms (AdaMDOS and AdaMDOF) with proven near-optimal convergence rates for nonconvex optimization tasks.

Findings

AdaMDOS achieves near-optimal $ ilde{O}( ext{epsilon}^{-3})$ sample complexity.

AdaMDOF attains near-optimal $O( ext{sqrt}(n) ext{epsilon}^{-2})$ sample complexity.

Experimental results confirm the efficiency of the proposed algorithms.

Abstract

Decentralized learning recently has received increasing attention in machine learning due to its advantages in implementation simplicity and system robustness, data privacy. Meanwhile, the adaptive gradient methods show superior performances in many machine learning tasks such as training neural networks. Although some works focus on studying decentralized optimization algorithms with adaptive learning rates, these adaptive decentralized algorithms still suffer from high sample complexity. To fill these gaps, we propose a class of faster adaptive decentralized algorithms (i.e., AdaMDOS and AdaMDOF) for distributed nonconvex stochastic and finite-sum optimization, respectively. Moreover, we provide a solid convergence analysis framework for our methods. In particular, we prove that our AdaMDOS obtains a near-optimal sample complexity of $\tilde{O}(\epsilon^{-3})$ for finding an $\epsilon$-stationary solution of nonconvex stochastic optimization. Meanwhile, our AdaMDOF obtains a near-optimal sample complexity of $O(\sqrt{n}\epsilon^{-2})$ for finding an $\epsilon$-stationary solution of nonconvex finite-sum optimization, where $n$ denotes the sample size. To the best of our knowledge, our AdaMDOF algorithm is the first adaptive decentralized algorithm for nonconvex finite-sum optimization. Some experimental results demonstrate efficiency of our algorithms.