An adaptive framework for first-order gradient methods

TL;DR

This paper introduces an adaptive framework for first-order gradient methods that dynamically adjusts step size and momentum without prior knowledge of the strong convexity parameter, improving convergence speed and practical efficiency.

Contribution

The paper presents a novel adaptive approach for gradient methods that estimates convergence rate empirically, enabling automatic parameter tuning and enhanced performance in optimization tasks.

Findings

Converges at least as fast as gradient descent on quadratic problems.

Effective in both quadratic and nonlinear optimization problems.

Comparable or improved performance over fixed-parameter methods.

Abstract

Gradient methods are widely used in optimization problems. In practice, while the smoothness parameter can be estimated utilizing techniques such as backtracking, estimating the strong convexity parameter remains a challenge; moreover, even with the optimal parameter choice, convergence can be slow. In this work, we propose a framework for dynamically adapting the step size and momentum parameters in first-order gradient methods for the optimization problem, without prior knowledge of the strong convexity parameter. The main idea is to use the geometric average of the ratios of successive residual norms as an empirical estimate of the upper bound on the convergence rate, which in turn allows us to adaptively update the algorithm parameters. The resulting algorithms are simple to implement, yet efficient in practice, requiring only a few additional computations on existing information. The proposed adaptive gradient methods are shown to converge at least as fast as gradient descent for quadratic optimization problems. Numerical experiments on both quadratic and nonlinear problems validate the effectiveness of the proposed adaptive algorithms. The results show that the adaptive algorithms are comparable to their counterparts using optimal parameters, and in some cases, they capture local information and exhibit improved performance.