AdaGrad-Diff: A New Version of the Adaptive Gradient Algorithm

TL;DR

This paper introduces AdaGrad-Diff, an adaptive gradient method that adjusts stepsizes based on the differences in successive gradients, improving robustness over traditional AdaGrad especially in settings with varying gradient behavior.

Contribution

The paper presents a novel adaptive algorithm that uses gradient differences for stepsize adjustment, enhancing stability and robustness compared to existing methods.

Findings

More robust than AdaGrad in practical scenarios

Automatically adjusts stepsize based on gradient variation

Effective in settings with curvature or instability

Abstract

Vanilla gradient methods are often highly sensitive to the choice of stepsize, which typically requires manual tuning. Adaptive methods alleviate this issue and have therefore become widely used. Among them, AdaGrad has been particularly influential. In this paper, we propose an AdaGrad-style adaptive method in which the adaptation is driven by the cumulative squared norms of successive gradient differences rather than gradient norms themselves. The key idea is that when gradients vary little across iterations, the stepsize is not unnecessarily reduced, while significant gradient fluctuations, reflecting curvature or instability, lead to automatic stepsize damping. Numerical experiments demonstrate that the proposed method is more robust than AdaGrad in several practically relevant settings.