Inference of Online Newton Methods with Nesterov's Accelerated Sketching
Haoxuan Wang, Xinchen Du, Sen Na

TL;DR
This paper introduces an efficient online second-order Newton method with Nesterov's accelerated sketching, enabling robust uncertainty quantification in streaming data scenarios with reduced computational complexity.
Contribution
It proposes a novel online Newton method using Nesterov's accelerated sketching to approximate Newton directions with $O(d^2)$ complexity, improving inference robustness.
Findings
The method achieves global almost-sure convergence.
Asymptotic normality of the last iterate is established.
Experiments show superior online inference performance.
Abstract
Reliable decision-making with streaming data requires principled uncertainty quantification of online methods. While first-order methods enable efficient iterate updates, their inference procedures still require updating proper (covariance) matrices, incurring time and memory complexity, and are sensitive to ill-conditioning and noise heterogeneity of the problem. This costly inference task offers an opportunity for more robust second-order methods, which are, however, bottlenecked by solving Newton systems with complexity. In this paper, we address this gap by studying an online Newton method with Hessian averaging, where the Newton direction at each step is approximately computed using a sketch-and-project solver with Nesterov's acceleration, matching complexity of first-order methods. For the proposed method, we quantify its uncertainty arising from both…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
