Towards Simple and Provable Parameter-Free Adaptive Gradient Methods

TL;DR

This paper introduces AdaGrad++ and Adam++, simple parameter-free variants of AdaGrad and Adam that come with formal convergence guarantees and perform competitively in deep learning tasks.

Contribution

It proposes novel, simple, parameter-free optimization algorithms with proven convergence guarantees for convex and deep learning applications.

Findings

AdaGrad++ achieves convergence rates comparable to AdaGrad.

Adam++ matches Adam's convergence rate without learning rate conditions.

Experimental results show competitive performance across deep learning tasks.

Abstract

Optimization algorithms such as AdaGrad and Adam have significantly advanced the training of deep models by dynamically adjusting the learning rate during the optimization process. However, adhoc tuning of learning rates poses a challenge, leading to inefficiencies in practice. To address this issue, recent research has focused on developing "learning-rate-free" or "parameter-free" algorithms that operate effectively without the need for learning rate tuning. Despite these efforts, existing parameter-free variants of AdaGrad and Adam tend to be overly complex and/or lack formal convergence guarantees. In this paper, we present AdaGrad++ and Adam++, novel and simple parameter-free variants of AdaGrad and Adam with convergence guarantees. We prove that AdaGrad++ achieves comparable convergence rates to AdaGrad in convex optimization without predefined learning rate assumptions. Similarly, Adam++ matches the convergence rate of Adam without relying on any conditions on the learning rates. Experimental results across various deep learning tasks validate the competitive performance of AdaGrad++ and Adam++.