Right Decisions from Wrong Predictions: A Mechanism Design Alternative to Individual Calibration

TL;DR

This paper introduces a mechanism that aligns forecasted utility with actual utility for decision-makers relying on imperfect probabilistic forecasts, using fair bets and online learning to prevent exploitation.

Contribution

It proposes a novel, sustainable mechanism based on fair bets and online learning to ensure truthful confidence in individual probabilistic predictions.

Findings

Mechanism guarantees no exploitation over time.

Application enables passengers to optimize travel plans confidently.

Demonstrates practical utility in airline delay predictions.

Abstract

Decision makers often need to rely on imperfect probabilistic forecasts. While average performance metrics are typically available, it is difficult to assess the quality of individual forecasts and the corresponding utilities. To convey confidence about individual predictions to decision-makers, we propose a compensation mechanism ensuring that the forecasted utility matches the actually accrued utility. While a naive scheme to compensate decision-makers for prediction errors can be exploited and might not be sustainable in the long run, we propose a mechanism based on fair bets and online learning that provably cannot be exploited. We demonstrate an application showing how passengers could confidently optimize individual travel plans based on flight delay probabilities estimated by an airline.