Mixability made efficient: Fast online multiclass logistic regression

TL;DR

This paper introduces an efficient method for online multiclass logistic regression that maintains the optimal regret bounds of mixability-based algorithms while significantly reducing computational complexity.

Contribution

The authors develop a new algorithm that achieves optimal regret in online multiclass logistic regression with practical computational efficiency.

Findings

Achieves optimal regret bounds similar to existing mixability-based methods.

Reduces computational complexity from prohibitive levels to practical levels.

Demonstrates effectiveness through theoretical analysis and empirical validation.

Abstract

Mixability has been shown to be a powerful tool to obtain algorithms with optimal regret. However, the resulting methods often suffer from high computational complexity which has reduced their practical applicability. For example, in the case of multiclass logistic regression, the aggregating forecaster (Foster et al. (2018)) achieves a regret of $O(\log(Bn))$ whereas Online Newton Step achieves $O(e^B\log(n))$ obtaining a double exponential gain in $B$ (a bound on the norm of comparative functions). However, this high statistical performance is at the price of a prohibitive computational complexity $O(n^{37})$.