Online Newton Step Algorithm with Estimated Gradient

TL;DR

This paper introduces ONSEG, a second-order online learning algorithm that leverages estimated gradients to achieve faster convergence in bandit feedback scenarios, improving regret bounds over previous methods.

Contribution

The paper develops ONSEG, extending the Online Newton Step algorithm with expected gradient estimation, reducing regret bounds from 0505/615 to 0502/315, and demonstrating empirical advantages.

Findings

ONSEG reduces expected regret from 0505/615 to 0502/315.

The algorithm outperforms existing methods on multiple real-world datasets.

Second-order information accelerates convergence in bandit online learning.

Abstract

Online learning with limited information feedback (bandit) tries to solve the problem where an online learner receives partial feedback information from the environment in the course of learning. Under this setting, Flaxman et al.[8] extended Zinkevich's classical Online Gradient Descent (OGD) algorithm [29] by proposing the Online Gradient Descent with Expected Gradient (OGDEG) algorithm. Specifically, it uses a simple trick to approximate the gradient of the loss function $f_t$ by evaluating it at a single point and bounds the expected regret as $\mathcal{O}(T^{5/6})$ [8], where the number of rounds is $T$. Meanwhile, past research efforts have shown that compared with the first-order algorithms, second-order online learning algorithms such as Online Newton Step (ONS) [11] can significantly accelerate the convergence rate of traditional online learning algorithms. Motivated by this, this paper aims to exploit the second-order information to speed up the convergence of the OGDEG algorithm. In particular, we extend the ONS algorithm with the trick of expected gradient and develop a novel second-order online learning algorithm, i.e., Online Newton Step with Expected Gradient (ONSEG). Theoretically, we show that the proposed ONSEG algorithm significantly reduces the expected regret of OGDEG algorithm from $\mathcal{O}(T^{5/6})$ to $\mathcal{O}(T^{2/3})$ in the bandit feedback scenario. Empirically, we further demonstrate the advantages of the proposed algorithm on multiple real-world datasets.