On Acceleration with Noise-Corrupted Gradients

TL;DR

This paper introduces a new accelerated optimization method called AGDP, analyzes its stability under noisy gradient conditions, and provides modifications to improve robustness and reduce noise-induced errors.

Contribution

The paper presents AGDP, a generalized accelerated method, and offers a theoretical analysis of its performance with noisy gradients, including practical modifications for noise reduction.

Findings

AGDP outperforms previous methods in noisy settings

Theoretical analysis clarifies noise-acceleration interaction

Modified AGDP reduces gradient noise errors

Abstract

Accelerated algorithms have broad applications in large-scale optimization, due to their generality and fast convergence. However, their stability in the practical setting of noise-corrupted gradient oracles is not well-understood. This paper provides two main technical contributions: (i) a new accelerated method AGDP that generalizes Nesterov's AGD and improves on the recent method AXGD (Diakonikolas & Orecchia, 2018), and (ii) a theoretical study of accelerated algorithms under noisy and inexact gradient oracles, which is supported by numerical experiments. This study leverages the simplicity of AGDP and its analysis to clarify the interaction between noise and acceleration and to suggest modifications to the algorithm that reduce the mean and variance of the error incurred due to the gradient noise.