Gradient Methods with Memory

TL;DR

This paper introduces gradient methods with memory that leverage previous iteration information to accelerate convergence in smooth convex minimization, combining improved theoretical guarantees with efficient computations.

Contribution

The paper proposes a novel class of gradient methods with memory for convex functions, achieving faster convergence and computational efficiency compared to standard gradient methods.

Findings

Methods outperform traditional gradient methods in oracle calls

New algorithms demonstrate improved convergence rates

Preliminary experiments confirm theoretical advantages

Abstract

In this paper, we consider gradient methods for minimizing smooth convex functions, which employ the information obtained at the previous iterations in order to accelerate the convergence towards the optimal solution. This information is used in the form of a piece-wise linear model of the objective function, which provides us with much better prediction abilities as compared with the standard linear model. To the best of our knowledge, this approach was never really applied in Convex Minimization to differentiable functions in view of the high complexity of the corresponding auxiliary problems. However, we show that all necessary computations can be done very efficiently. Consequently, we get new optimization methods, which are better than the usual Gradient Methods both in the number of oracle calls and in the computational time. Our theoretical conclusions are confirmed by preliminary computational experiments.