Statistical Inference in High-dimensional Generalized Linear Models with Streaming Data

TL;DR

This paper introduces an online debiased lasso method for real-time statistical inference in high-dimensional generalized linear models with streaming data, ensuring consistency and asymptotic normality.

Contribution

It develops a novel online inference technique that uses summary statistics and corrects approximation errors, enabling real-time analysis of streaming high-dimensional data.

Findings

The online debiased estimators are consistent and asymptotically normal.

Numerical experiments confirm the effectiveness of the proposed method.

Application to automotive crash data demonstrates practical utility.

Abstract

In this paper we develop an online statistical inference approach for high-dimensional generalized linear models with streaming data for real-time estimation and inference. We propose an online debiased lasso (ODL) method to accommodate the special structure of streaming data. ODL differs from offline debiased lasso in two important aspects. First, in computing the estimate at the current stage, it only uses summary statistics of the historical data. Second, in addition to debiasing an online lasso estimator, ODL corrects an approximation error term arising from nonlinear online updating with streaming data. We show that the proposed online debiased estimators for the GLMs are consistent and asymptotically normal. This result provides a theoretical basis for carrying out real-time interim statistical inference with streaming data. Extensive numerical experiments are conducted to evaluate the performance of the proposed ODL method. These experiments demonstrate the effectiveness of our algorithm and support the theoretical results. A streaming dataset from the National Automotive Sampling System-Crashworthiness Data System is analyzed to illustrate the application of the proposed method.