Accelerating Perturbed Stochastic Iterates in Asynchronous Lock-Free Optimization

TL;DR

This paper demonstrates that stochastic acceleration is achievable in asynchronous lock-free optimization using the perturbed iterate framework, achieving optimal gradient complexity and maintaining linear speed-up, with a novel accelerated SVRG variant.

Contribution

It introduces a new accelerated SVRG algorithm with sparse updates for asynchronous optimization, achieving optimal complexity and preserving linear speed-up.

Findings

Achieves stochastic acceleration in asynchronous lock-free optimization.

Requires the same linear speed-up condition as non-accelerated methods.

Empirical results confirm theoretical improvements.

Abstract

We show that stochastic acceleration can be achieved under the perturbed iterate framework (Mania et al., 2017) in asynchronous lock-free optimization, which leads to the optimal incremental gradient complexity for finite-sum objectives. We prove that our new accelerated method requires the same linear speed-up condition as the existing non-accelerated methods. Our core algorithmic discovery is a new accelerated SVRG variant with sparse updates. Empirical results are presented to verify our theoretical findings.