Efficient Second Order Online Learning by Sketching

TL;DR

This paper introduces Sketched Online Newton (SON), an efficient second order online learning algorithm that uses sketching techniques to achieve improved regret bounds and computational efficiency, especially for ill-conditioned data.

Contribution

The paper presents a novel sketching-based approach for second order online learning, significantly reducing computational complexity and enhancing performance for challenging data conditions.

Findings

SON achieves linear running time in dimension and sketch size.

Sparse sketching methods further reduce computation to feature sparsity.

The approach eliminates previous computational barriers in second order online learning.

Abstract

We propose Sketched Online Newton (SON), an online second order learning algorithm that enjoys substantially improved regret guarantees for ill-conditioned data. SON is an enhanced version of the Online Newton Step, which, via sketching techniques enjoys a running time linear in the dimension and sketch size. We further develop sparse forms of the sketching methods (such as Oja's rule), making the computation linear in the sparsity of features. Together, the algorithm eliminates all computational obstacles in previous second order online learning approaches.