A Full Adagrad algorithm with O(Nd) operations

TL;DR

This paper introduces an efficient recursive algorithm for Full AdaGrad that reduces computational complexity from quadratic to linear in the product of data and dimension, enabling scalable stochastic optimization.

Contribution

The paper presents a novel recursive estimation method for the inverse square root of the gradient covariance, improving the efficiency of full-matrix adaptive gradient algorithms.

Findings

Reduces computational complexity to O(Nd)

Demonstrates effective convergence rates

Shows improved scalability in numerical experiments

Abstract

A novel approach is given to overcome the computational challenges of the full-matrix Adaptive Gradient algorithm (Full AdaGrad) in stochastic optimization. By developing a recursive method that estimates the inverse of the square root of the covariance of the gradient, alongside a streaming variant for parameter updates, the study offers efficient and practical algorithms for large-scale applications. This innovative strategy significantly reduces the complexity and resource demands typically associated with full-matrix methods, enabling more effective optimization processes. Moreover, the convergence rates of the proposed estimators and their asymptotic efficiency are given. Their effectiveness is demonstrated through numerical studies.