Competing with wild prediction rules

TL;DR

This paper develops new online prediction algorithms that are competitive with irregular benchmark classes, extending previous Hilbert space methods to Banach spaces with regret bounds depending on the class's structure.

Contribution

It introduces Banach-space techniques to create prediction algorithms with regret bounds for irregular classes beyond Hilbert spaces.

Findings

Achieves regret bounds of O(N^(-1/p)) for p in [2,∞)

Extends prediction methods to non-Hilbert benchmark classes

Provides theoretical guarantees for irregular prediction rule classes

Abstract

We consider the problem of on-line prediction competitive with a benchmark class of continuous but highly irregular prediction rules. It is known that if the benchmark class is a reproducing kernel Hilbert space, there exists a prediction algorithm whose average loss over the first N examples does not exceed the average loss of any prediction rule in the class plus a "regret term" of O(N^(-1/2)). The elements of some natural benchmark classes, however, are so irregular that these classes are not Hilbert spaces. In this paper we develop Banach-space methods to construct a prediction algorithm with a regret term of O(N^(-1/p)), where p is in [2,infty) and p-2 reflects the degree to which the benchmark class fails to be a Hilbert space.