A Modern Introduction to Online Learning

TL;DR

This book provides a comprehensive, accessible introduction to online learning, covering algorithms, theory, and applications within the framework of Online Convex Optimization and regret minimization.

Contribution

It offers a modern, unified presentation of online learning algorithms, including adaptive, parameter-free methods, and extends to advanced topics like bandits and non-stationary regret analysis.

Findings

Introduces first- and second-order algorithms for convex losses

Discusses adaptive and parameter-free online learning methods

Explores applications beyond traditional online learning domains

Abstract

In this book, I introduce the basic concepts of Online Learning through the modern view of Online Convex Optimization. Here, online learning refers to the framework of regret minimization under worst-case assumptions. I present first-order and second-order algorithms for online learning with convex losses, in Euclidean and non-Euclidean settings. All the algorithms are clearly presented as instantiation of Online Mirror Descent or Follow-The-Regularized-Leader and their variants. Particular attention is given to the issue of tuning the parameters of the algorithms and learning in unbounded domains, through adaptive and parameter-free online learning algorithms. Non-convex losses are addressed through convex surrogate losses and randomization. The bandit setting is also briefly discussed, touching on the problem of adversarial and stochastic multi-armed bandits. Finally, I also cover advanced topics, including black-box reductions, saddle-point optimization, sequential investment, and non-stationary forms of regret analysis. The book concludes with a selection of applications of online learning to domains far from it, such as generalization theory and concentration inequalities. I tried to maintain an informal, but mathematically serious, tone throughout the book. No prior knowledge of convex analysis is required. Moreover, all the included proofs have been carefully chosen to be as simple and as short as possible. This also means that sometimes I have added one or two additional assumptions, just to simplify the proofs.