Dual Averaging Converges for Nonconvex Smooth Stochastic Optimization

TL;DR

This paper proves that stochastic dual averaging (SDA) converges for non-convex smooth stochastic optimization, matching the convergence rates of SGD, and introduces ADA-DA, a variant that adapts auto-scaling without prior noise knowledge.

Contribution

It provides the first complete convergence analysis of SDA in non-convex stochastic settings, bridging a significant theoretical gap.

Findings

SDA converges at a rate of O(1/T + σ log T/√T) in non-convex smooth stochastic optimization.

A reduction technique shows SDA as SGD applied to implicitly regularized objectives.

ADA-DA achieves the same convergence rate without needing noise variance knowledge.

Abstract

Dual averaging and gradient descent with their stochastic variants stand as the two canonical recipe books for first-order optimization: Every modern variant can be viewed as a descendant of one or the other. In the convex regime, these algorithms have been deeply studied, and we know that they are essentially equivalent in terms of theoretical guarantees. On the other hand, in the non-convex setting, the situation is drastically different: While we know that SGD can minimize the gradient of non-convex smooth functions, no finite-time complexity guarantee for Stochastic Dual Averaging (SDA) was known in the same setting. In this paper, we close this gap by a reduction that views SDA as SGD applied to a sequence of implicitly regularized objectives. We show that a tuned SDA exhibits a rate of convergence $\mathcal{O}(1 / T + \sigma \log T/ \sqrt{T})$, similar to that of SGD under the same assumptions. To our best knowledge, this is the first complete convergence theory for dual averaging on non-convex smooth stochastic problems without restrictive assumptions, closing a long-standing open problem in the field. Beyond the base algorithm, we also discuss ADA-DA, a variant that marries SDA with AdaGrad's auto-scaling, which achieves the same rate without requiring knowledge of the noise variance.