Online Adaptive Methods, Universality and Acceleration

TL;DR

This paper introduces a universal, adaptive optimization method that achieves accelerated convergence for smooth problems and standard rates for non-smooth and stochastic cases, unifying multiple settings.

Contribution

It presents the first method that adaptively handles smooth, non-smooth, and stochastic convex optimization without modifications.

Findings

Demonstrates accelerated convergence in smooth convex optimization

Achieves standard convergence rates in non-smooth and stochastic settings

Empirical results validate theoretical guarantees

Abstract

We present a novel method for convex unconstrained optimization that, without any modifications, ensures: (i) accelerated convergence rate for smooth objectives, (ii) standard convergence rate in the general (non-smooth) setting, and (iii) standard convergence rate in the stochastic optimization setting. To the best of our knowledge, this is the first method that simultaneously applies to all of the above settings. At the heart of our method is an adaptive learning rate rule that employs importance weights, in the spirit of adaptive online learning algorithms (Duchi et al., 2011; Levy, 2017), combined with an update that linearly couples two sequences, in the spirit of (Allen-Zhu and Orecchia, 2017). An empirical examination of our method demonstrates its applicability to the above mentioned scenarios and corroborates our theoretical findings.