CompAdaGrad: A Compressed, Complementary, Computationally-Efficient Adaptive Gradient Method

TL;DR

CompAdaGrad is a new adaptive gradient method that balances the theoretical benefits of full-matrix regularization with computational efficiency, performing well in high-dimensional online learning tasks.

Contribution

It introduces CompAdaGrad, which combines low-dimensional full-matrix regularization with diagonal regularization in the complementary space, improving performance over diagonal AdaGrad while maintaining linear complexity.

Findings

Outperforms diagonal AdaGrad in experiments

Maintains linear per-iteration complexity

Provides theoretical guarantees independent of regularizer choice

Abstract

The adaptive gradient online learning method known as AdaGrad has seen widespread use in the machine learning community in stochastic and adversarial online learning problems and more recently in deep learning methods. The method's full-matrix incarnation offers much better theoretical guarantees and potentially better empirical performance than its diagonal version; however, this version is computationally prohibitive and so the simpler diagonal version often is used in practice. We introduce a new method, CompAdaGrad, that navigates the space between these two schemes and show that this method can yield results much better than diagonal AdaGrad while avoiding the (effectively intractable) $O(n^3)$ computational complexity of full-matrix AdaGrad for dimension $n$. CompAdaGrad essentially performs full-matrix regularization in a low-dimensional subspace while performing diagonal regularization in the complementary subspace. We derive CompAdaGrad's updates for composite mirror descent in case of the squared $\ell_2$ norm and the $\ell_1$ norm, demonstrate that its complexity per iteration is linear in the dimension, and establish guarantees for the method independent of the choice of composite regularizer. Finally, we show preliminary results on several datasets.