Efficient improper learning for online logistic regression

TL;DR

This paper introduces an efficient improper online logistic regression algorithm that achieves logarithmic regret without exponential dependence on the radius B, improving computational feasibility over previous methods.

Contribution

The authors develop a new improper learning algorithm using regularized empirical risk minimization that avoids exponential constants and maintains logarithmic regret.

Findings

Achieves regret of O(B log(Bn))

Per-round complexity is O(d^2)

Outperforms proper algorithms with exponential constants

Abstract

We consider the setting of online logistic regression and consider the regret with respect to the 2-ball of radius B. It is known (see [Hazan et al., 2014]) that any proper algorithm which has logarithmic regret in the number of samples (denoted n) necessarily suffers an exponential multiplicative constant in B. In this work, we design an efficient improper algorithm that avoids this exponential constant while preserving a logarithmic regret. Indeed, [Foster et al., 2018] showed that the lower bound does not apply to improper algorithms and proposed a strategy based on exponential weights with prohibitive computational complexity. Our new algorithm based on regularized empirical risk minimization with surrogate losses satisfies a regret scaling as O(B log(Bn)) with a per-round time-complexity of order O(d^2).