Less Regret via Online Conditioning

TL;DR

This paper introduces an adaptive online gradient descent algorithm with per-coordinate learning rate adjustments, providing stronger regret bounds and competitive performance in large-scale machine learning tasks.

Contribution

It presents a novel online gradient descent method with diagonal preconditioning, improving regret bounds over standard approaches.

Findings

Stronger regret bounds than standard online gradient descent.

Competitive performance in large-scale machine learning experiments.

Effective per-coordinate learning rate adaptation.

Abstract

We analyze and evaluate an online gradient descent algorithm with adaptive per-coordinate adjustment of learning rates. Our algorithm can be thought of as an online version of batch gradient descent with a diagonal preconditioner. This approach leads to regret bounds that are stronger than those of standard online gradient descent for general online convex optimization problems. Experimentally, we show that our algorithm is competitive with state-of-the-art algorithms for large scale machine learning problems.