Efficient Second Order Online Learning by Sketching
Haipeng Luo, Alekh Agarwal, Nicolo Cesa-Bianchi, John Langford

TL;DR
This paper introduces Sketched Online Newton (SON), an efficient second order online learning algorithm that uses sketching techniques to achieve improved regret bounds and computational efficiency, especially for ill-conditioned data.
Contribution
The paper presents a novel sketching-based approach for second order online learning, significantly reducing computational complexity and enhancing performance for challenging data conditions.
Findings
SON achieves linear running time in dimension and sketch size.
Sparse sketching methods further reduce computation to feature sparsity.
The approach eliminates previous computational barriers in second order online learning.
Abstract
We propose Sketched Online Newton (SON), an online second order learning algorithm that enjoys substantially improved regret guarantees for ill-conditioned data. SON is an enhanced version of the Online Newton Step, which, via sketching techniques enjoys a running time linear in the dimension and sketch size. We further develop sparse forms of the sketching methods (such as Oja's rule), making the computation linear in the sparsity of features. Together, the algorithm eliminates all computational obstacles in previous second order online learning approaches.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Bandit Algorithms Research · Machine Learning and Algorithms · Data Stream Mining Techniques
