On Adaptive Stochastic Optimization for Streaming Data: A Newton's Method with O(dN) Operations

TL;DR

This paper proposes an adaptive stochastic Newton's method for streaming data that achieves second-order optimization benefits with the computational cost of first-order methods, improving performance on ill-conditioned problems.

Contribution

Introduction of an inversion-free adaptive Newton's method with $ ext{O}(dN)$ complexity suitable for streaming data, bridging the gap between first- and second-order methods.

Findings

Outperforms existing methods in complex covariance scenarios

Maintains $ ext{O}(dN)$ computational complexity

Effective in ill-conditioned and challenging initializations

Abstract

Stochastic optimization methods encounter new challenges in the realm of streaming, characterized by a continuous flow of large, high-dimensional data. While first-order methods, like stochastic gradient descent, are the natural choice, they often struggle with ill-conditioned problems. In contrast, second-order methods, such as Newton's methods, offer a potential solution, but their computational demands render them impractical. This paper introduces adaptive stochastic optimization methods that bridge the gap between addressing ill-conditioned problems while functioning in a streaming context. Notably, we present an adaptive inversion-free Newton's method with a computational complexity matching that of first-order methods, $\mathcal{O}(dN)$, where $d$ represents the number of dimensions/features, and $N$ the number of data. Theoretical analysis confirms their asymptotic efficiency, and empirical evidence demonstrates their effectiveness, especially in scenarios involving complex covariance structures and challenging initializations. In particular, our adaptive Newton's methods outperform existing methods, while maintaining favorable computational efficiency.