Adaptive and optimal online linear regression on $\ell^1$-balls

TL;DR

This paper develops adaptive online linear regression algorithms for $\, ext{l}^1$-balls, providing regret bounds that adapt to data and parameters, with a detailed analysis of different regimes based on dimension and time horizon.

Contribution

It introduces adaptive algorithms for online linear regression on $\, ext{l}^1$-balls with regret bounds that are nearly optimal and do not require prior knowledge of key parameters.

Findings

Regret bounds with optimal dependencies on $d$, $T$, $U$, $X$, and $Y$.

Identification of a regime transition around $d = rac{ oot{2} ext{T} U X}{2 Y}$.

Algorithms that adapt without prior knowledge of parameters while achieving near-optimal regret.

Abstract

We consider the problem of online linear regression on individual sequences. The goal in this paper is for the forecaster to output sequential predictions which are, after $T$ time rounds, almost as good as the ones output by the best linear predictor in a given $\ell^1$-ball in $\\R^d$. We consider both the cases where the dimension~$d$ is small and large relative to the time horizon $T$. We first present regret bounds with optimal dependencies on $d$, $T$, and on the sizes $U$, $X$ and $Y$ of the $\ell^1$-ball, the input data and the observations. The minimax regret is shown to exhibit a regime transition around the point $d = \sqrt{T} U X / (2 Y)$. Furthermore, we present efficient algorithms that are adaptive, \ie, that do not require the knowledge of $U$, $X$, $Y$, and $T$, but still achieve nearly optimal regret bounds.