Adaptive Accelerated (Extra-)Gradient Methods with Variance Reduction

TL;DR

This paper introduces two adaptive variance-reduced gradient methods for convex optimization that do not require prior knowledge of smoothness parameters, achieving optimal convergence rates and outperforming existing adaptive algorithms.

Contribution

The paper proposes novel adaptive VR algorithms, AdaVRAE and AdaVRAG, with improved convergence rates that do not depend on smoothness parameter tuning, matching best-known results.

Findings

Algorithms achieve optimal convergence rates for convex optimization.

Experimental results show superior performance over previous methods.

Methods do not require knowledge of the smoothness parameter.

Abstract

In this paper, we study the finite-sum convex optimization problem focusing on the general convex case. Recently, the study of variance reduced (VR) methods and their accelerated variants has made exciting progress. However, the step size used in the existing VR algorithms typically depends on the smoothness parameter, which is often unknown and requires tuning in practice. To address this problem, we propose two novel adaptive VR algorithms: Adaptive Variance Reduced Accelerated Extra-Gradient (AdaVRAE) and Adaptive Variance Reduced Accelerated Gradient (AdaVRAG). Our algorithms do not require knowledge of the smoothness parameter. AdaVRAE uses $\mathcal{O}\left(n\log\log n+\sqrt{\frac{n\beta}{\epsilon}}\right)$ gradient evaluations and AdaVRAG uses $\mathcal{O}\left(n\log\log n+\sqrt{\frac{n\beta\log\beta}{\epsilon}}\right)$ gradient evaluations to attain an $\mathcal{O}(\epsilon)$-suboptimal solution, where $n$ is the number of functions in the finite sum and $\beta$ is the smoothness parameter. This result matches the best-known convergence rate of non-adaptive VR methods and it improves upon the convergence of the state of the art adaptive VR method, AdaSVRG. We demonstrate the superior performance of our algorithms compared with previous methods in experiments on real-world datasets.