LyAm: Robust Non-Convex Optimization for Stable Learning in Noisy Environments

TL;DR

LyAm is a new optimizer that combines Adam with Lyapunov stability theory to improve convergence robustness and stability in training deep neural networks on noisy, non-convex problems.

Contribution

It introduces LyAm, a novel optimizer integrating Lyapunov stability with Adam, providing theoretical convergence guarantees and improved empirical performance.

Findings

LyAm outperforms existing optimizers in accuracy.

LyAm achieves faster convergence.

LyAm enhances training stability in noisy environments.

Abstract

Training deep neural networks, particularly in computer vision tasks, often suffers from noisy gradients and unstable convergence, which hinder performance and generalization. In this paper, we propose LyAm, a novel optimizer that integrates Adam's adaptive moment estimation with Lyapunov-based stability mechanisms. LyAm dynamically adjusts the learning rate using Lyapunov stability theory to enhance convergence robustness and mitigate training noise. We provide a rigorous theoretical framework proving the convergence guarantees of LyAm in complex, non-convex settings. Extensive experiments on like as CIFAR-10 and CIFAR-100 show that LyAm consistently outperforms state-of-the-art optimizers in terms of accuracy, convergence speed, and stability, establishing it as a strong candidate for robust deep learning optimization.