Statistical Inference for the Population Landscape via Moment Adjusted Stochastic Gradients

TL;DR

This paper introduces moment-adjusted stochastic gradient methods that improve statistical inference by providing uncertainty quantification and acceleration effects in both convex and non-convex optimization problems.

Contribution

It proposes a novel stochastic optimization technique that accounts for data randomness, enabling better statistical inference and model mis-specification robustness.

Findings

Establishes non-asymptotic distributional theory for the proposed method.

Demonstrates acceleration effects similar to optimization techniques in generalized linear models.

Shows effectiveness in non-convex settings through numerical experiments.

Abstract

Modern statistical inference tasks often require iterative optimization methods to compute the solution. Convergence analysis from an optimization viewpoint only informs us how well the solution is approximated numerically but overlooks the sampling nature of the data. In contrast, recognizing the randomness in the data, statisticians are keen to provide uncertainty quantification, or confidence, for the solution obtained using iterative optimization methods. This paper makes progress along this direction by introducing the moment-adjusted stochastic gradient descents, a new stochastic optimization method for statistical inference. We establish non-asymptotic theory that characterizes the statistical distribution for certain iterative methods with optimization guarantees. On the statistical front, the theory allows for model mis-specification, with very mild conditions on the data. For optimization, the theory is flexible for both convex and non-convex cases. Remarkably, the moment-adjusting idea motivated from "error standardization" in statistics achieves a similar effect as acceleration in first-order optimization methods used to fit generalized linear models. We also demonstrate this acceleration effect in the non-convex setting through numerical experiments.