Parameter-free Clipped Gradient Descent Meets Polyak

TL;DR

This paper introduces a parameter-free clipped gradient descent method that adaptively adjusts hyperparameters, eliminating the need for tuning and maintaining convergence efficiency across various models and loss functions.

Contribution

It proposes the Inexact Polyak Stepsize, a novel hyperparameter-free approach for clipped gradient descent with proven convergence properties.

Findings

Converges to the optimal solution without hyperparameter tuning.

Effective in training models like LSTM, Nano-GPT, and T5.

Convergence rate is asymptotically independent of L under certain smoothness assumptions.

Abstract

Gradient descent and its variants are de facto standard algorithms for training machine learning models. As gradient descent is sensitive to its hyperparameters, we need to tune the hyperparameters carefully using a grid search. However, the method is time-consuming, particularly when multiple hyperparameters exist. Therefore, recent studies have analyzed parameter-free methods that adjust the hyperparameters on the fly. However, the existing work is limited to investigations of parameter-free methods for the stepsize, and parameter-free methods for other hyperparameters have not been explored. For instance, although the gradient clipping threshold is a crucial hyperparameter in addition to the stepsize for preventing gradient explosion issues, none of the existing studies have investigated parameter-free methods for clipped gradient descent. Therefore, in this study, we investigate the parameter-free methods for clipped gradient descent. Specifically, we propose Inexact Polyak Stepsize, which converges to the optimal solution without any hyperparameters tuning, and its convergence rate is asymptotically independent of $L$ under $L$-smooth and $(L_0, L_1)$-smooth assumptions of the loss function, similar to that of clipped gradient descent with well-tuned hyperparameters. We numerically validated our convergence results using a synthetic function and demonstrated the effectiveness of our proposed methods using LSTM, Nano-GPT, and T5.