Factor Augmented High-Dimensional SGD

TL;DR

This paper introduces Factor-Augmented SGD, a scalable streaming optimization method that incorporates latent factor estimation, with theoretical guarantees for high-dimensional machine learning tasks.

Contribution

It proposes a novel factor-augmented SGD algorithm that operates on streaming data and provides the first theoretical analysis including latent factor estimation error.

Findings

Operates purely on streaming data, scalable to large high-dimensional problems.

Provides moment convergence analysis under decaying step sizes and mini-batch updates.

Establishes a new theoretical framework incorporating latent factor estimation error.

Abstract

Stochastic gradient descent (SGD) is a fundamental optimization algorithm widely used in modern machine learning. In this paper, we propose Factor-Augmented SGD (FSGD), a new optimization method that leverages latent factor representations in high-dimensional learning tasks. Unlike standard two-stage dimension reduction approaches that rely on offline representation learning and full data storage, a key novelty of FSGD is that it operates purely on streaming data, making it scalable to large-scale and high-dimensional problems. Furthermore, we establish the first theoretical framework that explicitly incorporates latent factor estimation error into the analysis of SGD, and provide moment convergence in $\ell^s$ norm under decaying step sizes and mini-batch updates. Our results provide a new foundation for employing SGD reliably and scalably in high-dimensional machine learning systems.