SHANG++: Robust Stochastic Acceleration under Multiplicative Noise

TL;DR

This paper introduces SHANG++, a robust accelerated stochastic gradient method designed to handle multiplicative noise effectively, with proven convergence and strong empirical performance in deep learning tasks.

Contribution

The paper develops SHANG++, a new accelerated stochastic gradient method with enhanced noise robustness and faster convergence, improving stability under multiplicative noise conditions.

Findings

SHANG++ achieves faster convergence and better noise robustness than existing methods.

In experiments, SHANG++ maintains high accuracy with minimal hyperparameter tuning.

SHANG++ performs well on deep learning tasks, including ResNet-34, under noisy conditions.

Abstract

Under the multiplicative noise scaling (MNS) condition, original Nesterov acceleration is provably sensitive to noise and may diverge when gradient noise overwhelms the signal. In this paper, we develop two accelerated stochastic gradient descent methods by discretizing the Hessian-driven Nesterov accelerated gradient flow. We first derive SHANG, a direct Gauss-Seidel-type discretization that already improves stability under MNS. We then introduce SHANG++, which adds a damping correction and achieves faster convergence with stronger noise robustness. We establish convergence guarantees for both convex and strongly convex objectives under MNS, together with explicit parameter choices. In our experiments, SHANG++ performs consistently well across convex problems and applications in deep learning. In a dedicated noise experiment on ResNet-34, a single hyperparameter configuration attains accuracy within 1% of the noise-free setting. Across all experiments, SHANG++ outperforms existing accelerated methods in robustness and efficiency, with minimal parameter sensitivity.