Global Stability and Step Size Robustness of RMSProp
Naum Dimitrieski, Maria Christine Honecker, Carsten Scherer, Christian Ebenbauer
TL;DR
This paper introduces a Lyapunov function for RMSProp, proving its global stability and robustness to varying step sizes, enhancing understanding of its convergence behavior.
Contribution
It provides the first rigorous stability analysis of RMSProp with respect to constant and time-varying step sizes.
Findings
Proves global asymptotic stability of RMSProp with fixed step sizes.
Establishes robustness of RMSProp to bounded, time-varying step size rules.
Abstract
In this paper, an input-to-state Lyapunov function for the RMSProp optimization algorithm is introduced. Global asymptotic stability of the RMSProp algorithm for constant step sizes and robustness properties with respect to arbitrary bounded time-varying step size rules are established.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Optimization Algorithms Research · Distributed Control Multi-Agent Systems · Advanced Control Systems Optimization