A More Stable Accelerated Gradient Method Inspired by Continuous-Time Perspective

TL;DR

This paper introduces a new, more stable accelerated gradient method inspired by continuous-time analysis, improving upon Nesterov's method by enhancing stability and computational efficiency in machine learning tasks.

Contribution

The paper proposes a higher-order accelerated gradient method with improved stability based on numerical analysis of its continuous-time limit, outperforming NAG in stability and speed.

Findings

The new method is more stable than NAG for large step sizes.

Experiments show improved stability in matrix completion and digit recognition.

Enhanced stability leads to faster computation in practice.

Abstract

Nesterov's accelerated gradient method (NAG) is widely used in problems with machine learning background including deep learning, and is corresponding to a continuous-time differential equation. From this connection, the property of the differential equation and its numerical approximation can be investigated to improve the accelerated gradient method. In this work we present a new improvement of NAG in terms of stability inspired by numerical analysis. We give the precise order of NAG as a numerical approximation of its continuous-time limit and then present a new method with higher order. We show theoretically that our new method is more stable than NAG for large step size. Experiments of matrix completion and handwriting digit recognition demonstrate that the stability of our new method is better. Furthermore, better stability leads to higher computational speed in experiments.