Nonasymptotic Analysis of Accelerated Methods With Inexact Oracle Under Absolute Error Bound

TL;DR

This paper provides nonasymptotic convergence bounds for accelerated first-order methods with inexact gradients, using performance estimation problems to optimize error management and stepsize strategies in convex optimization.

Contribution

It introduces novel convergence bounds for GOGM and GFGM with inexact oracles, optimizing error bounds and stepsize strategies based on analytical solutions to PEP.

Findings

Derived convergence bounds that are independent of initial conditions.

Identified optimal strategies for setting gradient inexactness.

Analyzed the tradeoff between error accumulation and convergence speed.

Abstract

Performance analysis of first-order algorithms with inexact oracles has gained recent attention due to various emerging applications in which obtaining exact gradients is impossible or computationally expensive. Previous research has demonstrated that the performance of accelerated first-order methods is more sensitive to gradient errors compared with non-accelerated ones. This paper investigates the nonasymptotic convergence bound of two accelerated methods with inexact gradients to solve deterministic smooth convex problems. Performance Estimation Problem (PEP) is used as the primary tool to analyze the convergence bounds of the underlying algorithms. By finding an analytical solution to PEP, we derive novel convergence bounds of Generalized Optimized Gradient Method (GOGM) and Generalized Fast Gradient Method (GFGM) with inexact gradient oracles following the absolute error bound. The derived bounds allow varying oracle inexactness along the iterations; furthermore, their accumulated error terms are independent of the initial condition and any unknown parameters. Furthermore, we analyze the tradeoff between the vanishing term and the accumulated error in the convergence bound that guides finding the optimal stepsize. Finally, we determine the optimal strategy to set the gradient inexactness along iterations (if possible in a given application), ensuring that the accumulated error remains subordinate to the vanishing term.