DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Method

TL;DR

This paper introduces DoWG, a parameter-free gradient descent optimizer that adapts to various problem types, matching the efficiency of tuned methods without requiring parameter tuning, and demonstrates strong empirical performance.

Contribution

The paper presents DoWG, a novel universal and parameter-free optimizer that maintains a distance-based weighted average of gradients, achieving optimal convergence rates without tuning.

Findings

Matches convergence rate of optimally tuned gradient descent in convex problems

Automatically adapts to smooth and nonsmooth problems

Effective in practical machine learning tasks

Abstract

This paper proposes a new easy-to-implement parameter-free gradient-based optimizer: DoWG (Distance over Weighted Gradients). We prove that DoWG is efficient -- matching the convergence rate of optimally tuned gradient descent in convex optimization up to a logarithmic factor without tuning any parameters, and universal -- automatically adapting to both smooth and nonsmooth problems. While popular algorithms following the AdaGrad framework compute a running average of the squared gradients to use for normalization, DoWG maintains a new distance-based weighted version of the running average, which is crucial to achieve the desired properties. To complement our theory, we also show empirically that DoWG trains at the edge of stability, and validate its effectiveness on practical machine learning tasks.