Online Prediction with Limited Selectivity

TL;DR

This paper introduces a new model called Prediction with Limited Selectivity (PLS), where forecasters can only start predictions at certain times, and analyzes the optimal prediction error in this constrained setting.

Contribution

It formalizes the PLS model, develops an instance-dependent complexity measure, and provides bounds on the optimal error that hold with high probability for random instances.

Findings

Instance-dependent bounds on prediction error

High-probability bounds for random instances

Analysis of optimal prediction error under selectivity constraints

Abstract

Selective prediction [Dru13, QV19] models the scenario where a forecaster freely decides on the prediction window that their forecast spans. Many data statistics can be predicted to a non-trivial error rate without any distributional assumptions or expert advice, yet these results rely on that the forecaster may predict at any time. We introduce a model of Prediction with Limited Selectivity (PLS) where the forecaster can start the prediction only on a subset of the time horizon. We study the optimal prediction error both on an instance-by-instance basis and via an average-case analysis. We introduce a complexity measure that gives instance-dependent bounds on the optimal error. For a randomly-generated PLS instance, these bounds match with high probability.