Perturbed Gradient Descent Algorithms are Small-Disturbance Input-to-State Stable

TL;DR

This paper introduces a robustness framework for gradient descent algorithms under perturbations, showing they are stable and converge near optima if the objective satisfies a generalized nonlinear PL condition, with applications to LQR and policy gradients.

Contribution

It extends the linear Polyak-Lojasiewicz condition to a nonlinear version and proves small-disturbance input-to-state stability for various policy gradient algorithms.

Findings

Gradient descent is small-disturbance ISS under nonlinear PL condition.

LQR cost satisfies the nonlinear PL condition, ensuring policy gradient stability.

Natural policy gradient and Gauss-Newton methods are also proven to be small-disturbance ISS.

Abstract

This article investigates the robustness of gradient descent algorithms under perturbations. The concept of small-disturbance input-to-state stability (ISS) for discrete-time nonlinear dynamical systems is introduced, along with its Lyapunov characterization. The conventional linear Polyak-Lojasiewicz (PL) condition is then extended to a nonlinear version, and it is shown that the gradient descent algorithm is small-disturbance ISS provided the objective function satisfies the generalized nonlinear PL condition. This small-disturbance ISS property guarantees that the gradient descent algorithm converges to a small neighborhood of the optimum under sufficiently small perturbations. As a direct application of the developed framework, we demonstrate that the LQR cost satisfies the generalized nonlinear PL condition, thereby establishing that the policy gradient algorithm for LQR is small-disturbance ISS. Additionally, other popular policy gradient algorithms, including natural policy gradient and Gauss-Newton method, are also proven to be small-disturbance ISS.