AdaGrad under Anisotropic Smoothness

TL;DR

This paper introduces a new anisotropic smoothness assumption to analyze AdaGrad, demonstrating its faster convergence in large-scale deep learning tasks under realistic conditions, bridging the gap between theory and practice.

Contribution

It proposes a novel anisotropic smoothness framework and provides theoretical analysis showing AdaGrad's improved convergence guarantees in large batch settings.

Findings

AdaGrad achieves better dimensional dependence under anisotropic smoothness.

Experimental results support the new smoothness assumption and theoretical findings.

Analysis applies to logistic regression and instruction fine-tuning tasks.

Abstract

Adaptive gradient methods have been widely adopted in training large-scale deep neural networks, especially large foundation models. Despite the huge success in practice, their theoretical advantages over classical gradient methods with uniform step sizes across all coordinates (e.g. SGD) have not been fully understood, especially in the large batch-size setting commonly used in practice. This is because the only theoretical result that can demonstrate this benefit was obtained in the original paper of Adagrad for convex nonsmooth objective functions, which is insufficient for large batch algorithms. In this work, we attempt to resolve this gap between theory and practice by proposing a novel anisotropic generalized smoothness assumption and providing corresponding analyses of Adagrad. It is shown that under anisotropic smoothness and noise conditions, AdaGrad can achieve faster convergence guarantees in terms of better dimensional dependence than algorithms with uniform step sizes across all coordinates. Experiments in logistic regression and instruction following fine-tuning tasks provide strong evidence to support our novel assumption and theoretical analysis.