"Calibeating": Beating Forecasters at Their Own Game
Dean P. Foster, Sergiu Hart

TL;DR
This paper introduces methods to outperform forecasters' calibration scores without sacrificing their expertise, using deterministic and stochastic procedures to achieve 'calibeating' and extending these results to multiple and continuous calibration scenarios.
Contribution
It presents novel online algorithms for calibeating forecasters, including deterministic and stochastic methods, and extends the concept to multiple and continuously calibrated procedures.
Findings
Deterministic online procedure for calibeating any forecast.
Stochastic procedure that is itself calibrated and achieves calibeating.
Extension to multiple procedures and continuous calibration scenarios.
Abstract
In order to identify expertise, forecasters should not be tested by their calibration score, which can always be made arbitrarily small, but rather by their Brier score. The Brier score is the sum of the calibration score and the refinement score; the latter measures how good the sorting into bins with the same forecast is, and thus attests to "expertise." This raises the question of whether one can gain calibration without losing expertise, which we refer to as "calibeating." We provide an easy way to calibeat any forecast, by a deterministic online procedure. We moreover show that calibeating can be achieved by a stochastic procedure that is itself calibrated, and then extend the results to simultaneously calibeating multiple procedures, and to deterministic procedures that are continuously calibrated.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
"Calibeating": Beating Forecasters at Their Own Game· youtube
Taxonomy
TopicsForecasting Techniques and Applications · Monetary Policy and Economic Impact · Complex Systems and Time Series Analysis
