Non asymptotic analysis of Adaptive stochastic gradient algorithms and applications

TL;DR

This paper provides a non-asymptotic theoretical analysis of adaptive stochastic gradient algorithms like Adagrad and Stochastic Newton, specifically for strongly convex problems, with applications to linear regression and generalized linear models.

Contribution

It offers the first non-asymptotic analysis of adaptive gradient algorithms for strongly convex objectives, extending theoretical understanding beyond classical methods.

Findings

Theoretical bounds for Adagrad and Stochastic Newton algorithms.

Applications to linear regression and generalized linear models.

Insights into algorithm performance in ill-conditioned problems.

Abstract

In stochastic optimization, a common tool to deal sequentially with large sample is to consider the well-known stochastic gradient algorithm. Nevertheless, since the stepsequence is the same for each direction, this can lead to bad results in practice in case of ill-conditionned problem. To overcome this, adaptive gradient algorithms such that Adagrad or Stochastic Newton algorithms should be prefered. This paper is devoted to the non asymptotic analyis of these adaptive gradient algorithms for strongly convex objective. All the theoretical results will be adapted to linear regression and regularized generalized linear model for both Adagrad and Stochastic Newton algorithms.