Trading-Off Static and Dynamic Regret in Online Least-Squares and Beyond

TL;DR

This paper rigorously analyzes the impact of forgetting factors in online Newton algorithms for non-stationary data, establishing optimal dynamic regret bounds and proposing a new gradient descent step size rule for efficiency.

Contribution

It provides a theoretical characterization of forgetting factors in online Newton algorithms and introduces a novel gradient descent step size rule for strongly convex functions.

Findings

Achieves dynamic regret bounds of O(log T) and O(√TV) for exp-concave and strongly convex objectives.

Classic recursive least-squares with a forgetting factor attains these regret bounds.

New step size rule improves computational efficiency while maintaining optimal regret bounds.

Abstract

Recursive least-squares algorithms often use forgetting factors as a heuristic to adapt to non-stationary data streams. The first contribution of this paper rigorously characterizes the effect of forgetting factors for a class of online Newton algorithms. For exp-concave and strongly convex objectives, the algorithms achieve the dynamic regret of $\max\{O(\log T),O(\sqrt{TV})\}$, where $V$ is a bound on the path length of the comparison sequence. In particular, we show how classic recursive least-squares with a forgetting factor achieves this dynamic regret bound. By varying $V$, we obtain a trade-off between static and dynamic regret. In order to obtain more computationally efficient algorithms, our second contribution is a novel gradient descent step size rule for strongly convex functions. Our gradient descent rule recovers the order optimal dynamic regret bounds described above. For smooth problems, we can also obtain static regret of $O(T^{1-\beta})$ and dynamic regret of $O(T^\beta V^*)$, where $\beta \in (0,1)$ and $V^*$ is the path length of the sequence of minimizers. By varying $\beta$, we obtain a trade-off between static and dynamic regret.