On Scalable Inference with Stochastic Gradient Descent

TL;DR

This paper introduces a scalable inference method for stochastic gradient descent that updates estimates with each new data point and perturbed estimates, enabling practical statistical inference for large datasets.

Contribution

It proposes a novel, scalable inferential procedure for SGD that is easy to implement and applicable to a wide range of models, including GLMs and quantile regression.

Findings

Method performs well in simulations.

Applicable to large datasets and online updating.

Theoretical guarantees established.

Abstract

In many applications involving large dataset or online updating, stochastic gradient descent (SGD) provides a scalable way to compute parameter estimates and has gained increasing popularity due to its numerical convenience and memory efficiency. While the asymptotic properties of SGD-based estimators have been established decades ago, statistical inference such as interval estimation remains much unexplored. The traditional resampling method such as the bootstrap is not computationally feasible since it requires to repeatedly draw independent samples from the entire dataset. The plug-in method is not applicable when there are no explicit formulas for the covariance matrix of the estimator. In this paper, we propose a scalable inferential procedure for stochastic gradient descent, which, upon the arrival of each observation, updates the SGD estimate as well as a large number of randomly perturbed SGD estimates. The proposed method is easy to implement in practice. We establish its theoretical properties for a general class of models that includes generalized linear models and quantile regression models as special cases. The finite-sample performance and numerical utility is evaluated by simulation studies and two real data applications.