Optimal Asynchronous Stochastic Nonconvex Optimization under Heavy-Tailed Noise

TL;DR

This paper introduces an asynchronous normalized stochastic gradient descent algorithm with momentum for nonconvex optimization under heavy-tailed noise, achieving optimal time complexity and demonstrating effectiveness through experiments.

Contribution

It presents a novel asynchronous optimization method that handles heavy-tailed noise and heterogeneity, with proven optimal time complexity under certain moment conditions.

Findings

Achieves optimal time complexity for heavy-tailed noise scenarios.

Demonstrates effectiveness through numerical experiments.

Handles arbitrarily heterogeneous computation times.

Abstract

This paper considers the problem of asynchronous stochastic nonconvex optimization with heavy-tailed gradient noise and arbitrarily heterogeneous computation times across workers. We propose an asynchronous normalized stochastic gradient descent algorithm with momentum. The analysis show that our method achieves the optimal time complexity under the assumption of bounded $p$th-order central moment with $p\in(1,2]$. We also provide numerical experiments to show the effectiveness of proposed method.