Gradient-Variation Online Adaptivity for Accelerated Optimization with H\"older Smoothness

TL;DR

This paper introduces an adaptive online learning algorithm for H"older smooth functions that seamlessly transitions between smooth and non-smooth regimes, leading to a universal accelerated optimization method.

Contribution

It develops a gradient-variation online learning algorithm that adapts to H"older smoothness without prior knowledge, and extends this to a universal offline optimization method with accelerated convergence.

Findings

The online algorithm interpolates regret bounds between smooth and non-smooth cases.

The offline method achieves accelerated convergence in smooth regimes.

The approach is fully adaptive and does not require prior smoothness knowledge.

Abstract

Smoothness is known to be crucial for acceleration in offline optimization, and for gradient-variation regret minimization in online learning. Interestingly, these two problems are actually closely connected -- accelerated optimization can be understood through the lens of gradient-variation online learning. In this paper, we investigate online learning with H\"older smooth functions, a general class encompassing both smooth and non-smooth (Lipschitz) functions, and explore its implications for offline optimization. For (strongly) convex online functions, we design the corresponding gradient-variation online learning algorithm whose regret smoothly interpolates between the optimal guarantees in smooth and non-smooth regimes. Notably, our algorithms do not require prior knowledge of the H\"older smoothness parameter, exhibiting strong adaptivity over existing methods. Through online-to-batch conversion, this gradient-variation online adaptivity yields an optimal universal method for stochastic convex optimization under H\"older smoothness. However, achieving universality in offline strongly convex optimization is more challenging. We address this by integrating online adaptivity with a detection-based guess-and-check procedure, which, for the first time, yields a universal offline method that achieves accelerated convergence in the smooth regime while maintaining near-optimal convergence in the non-smooth one.