High-dimensional online learning via asynchronous decomposition: Non-divergent results, dynamic regularization, and beyond

TL;DR

This paper introduces an asynchronous decomposition framework with dynamic regularization for high-dimensional online learning, ensuring non-divergent error bounds and adaptive accuracy in sparse settings.

Contribution

It proposes a novel asynchronous decomposition method with a dynamic-regularized algorithm that guarantees stable error bounds and sparsity, advancing high-dimensional online learning.

Findings

Maintains non-divergent error bounds across batches

Achieves oracle accuracy as data accumulates

Efficient in computation and memory for sparse optimization

Abstract

Existing high-dimensional online learning methods often face the challenge that their error bounds, or per-batch sample sizes, diverge as the number of data batches increases. To address this issue, we propose an asynchronous decomposition framework that leverages summary statistics to construct a surrogate score function for current-batch learning. This framework is implemented via a dynamic-regularized iterative hard thresholding algorithm, providing a computationally and memory-efficient solution for sparse online optimization. We provide a unified theoretical analysis that accounts for both the streaming computational error and statistical accuracy, establishing that our estimator maintains non-divergent error bounds and $\ell_0$ sparsity across all batches. Furthermore, the proposed estimator adaptively achieves additional gains as batches accumulate, attaining the oracle accuracy as if the entire historical dataset were accessible and the true support were known. These theoretical properties are further illustrated through an example of the generalized linear model.