Convergence of Batch Updating Methods with Approximate Gradients and/or Noisy Measurements: Theory and Computational Results

TL;DR

This paper introduces a comprehensive framework for analyzing batch updating methods in high-dimensional optimization, accommodating noisy and approximate gradients, and unifies existing convergence results for various approaches.

Contribution

It provides a unified theoretical framework applicable to convex and nonconvex functions, including noisy and approximate gradient scenarios, extending the understanding of convergence in optimization algorithms.

Findings

Framework encompasses all current batch updating methods

Establishes general convergence theorem for stochastic processes

Numerical experiments show potential failure of momentum methods with approximate gradients

Abstract

In this paper, we present a unified and general framework for analyzing the batch updating approach to nonlinear, high-dimensional optimization. The framework encompasses all the currently used batch updating approaches, and is applicable to nonconvex as well as convex functions. Moreover, the framework permits the use of noise-corrupted gradients, as well as first-order approximations to the gradient (sometimes referred to as "gradient-free" approaches). By viewing the analysis of the iterations as a problem in the convergence of stochastic processes, we are able to establish a very general theorem, which includes most known convergence results for zeroth-order and first-order methods. The analysis of "second-order" or momentum-based methods is not a part of this paper, and will be studied elsewhere. However, numerical experiments indicate that momentum-based methods can fail if the true gradient is replaced by its first-order approximation. This requires further theoretical analysis.