Lower and upper bounds of the convergence rate of gradient methods with composite noise in gradient

TL;DR

This paper analyzes the convergence of first-order optimization methods under composite gradient noise, proposing algorithms and bounds that account for practical inexact gradient scenarios such as biased compressors and floating-point errors.

Contribution

It introduces an algorithm that optimally handles absolute error accumulation and provides convergence bounds considering composite noise, with adaptive gradient descent and restart techniques.

Findings

Convergence bounds depend on the relative and absolute noise components.

An adaptive gradient descent algorithm is proposed for relative error.

Lower bounds confirm the influence of noise parameters on convergence.

Abstract

We introduce a detailed analysis of the convergence of first-order methods with composite noise (sum of relative and absolute) in gradient for convex and smooth function minimization. This paper illustrates instances of practical problems where the utilization of inexact oracles becomes necessary, such as biased compressors, use of floating-point arithmetic and gradient-free optimization. We propose an algorithm that optimally accumulates absolute error, with intermediate convergence depending on the relative component of the noise. Usage of restart technique, regularization transformation, and stopping criteria has been demonstrated to yield results for various function classes. Also, gradient descent adaptive to relative error parameter is provided. For relative noise, lower bounds of convergence are given, confirming the dependence of the parameter of the noise on the condition number of the problem.