Intermediate Gradient Methods with Relative Inexactness

TL;DR

This paper analyzes the robustness of accelerated gradient methods under relative inexactness in gradient information, proposing new bounds and an adaptive method that interpolates between different convergence regimes.

Contribution

It introduces a novel analysis of accelerated gradient methods with relative inexactness, establishing optimal bounds and proposing an adaptive intermediate method.

Findings

Accelerated methods tolerate larger relative errors while maintaining linear convergence.

The bounds on maximum admissible error are improved and shown to be optimal.

An adaptive intermediate gradient method is proposed for better robustness and convergence.

Abstract

This paper is devoted to first-order algorithms for smooth convex optimization with inexact gradients. Unlike the majority of the literature on this topic, we consider the setting of relative rather than absolute inexactness. More precisely, we assume that an additive error in the gradient is proportional to the gradient norm, rather than being globally bounded by some small quantity. We propose a novel analysis of the accelerated gradient method under relative inexactness and strong convexity and improve the bound on the maximum admissible error that preserves the linear convergence of the algorithm. In other words, we analyze how robust is the accelerated gradient method to the relative inexactness of the gradient information. Moreover, based on the Performance Estimation Problem (PEP) technique, we show that the obtained result is optimal for the family of accelerated algorithms we consider. Motivated by the existing intermediate methods with absolute error, i.e., the methods with convergence rates that interpolate between slower but more robust non-accelerated algorithms and faster, but less robust accelerated algorithms, we propose an adaptive variant of the intermediate gradient method with relative error in the gradient.