Conditional Forecasts and Proper Scoring Rules for Reliable and Accurate Performative Predictions

TL;DR

This paper investigates the challenges of performative predictions where forecasts influence outcomes, revealing fundamental limitations of classical scoring rules and proposing new methods for reliable, incentive-compatible forecasting and parameter estimation.

Contribution

It introduces conditions under which forecasts are invariant and proposes novel scoring rules and decision-theoretic solutions for eliciting correct, performatively stable forecasts.

Findings

Classical proper scoring rules fail in performative settings.

Separating covariates ensure well-posed forecasting problems.

New scoring methods enable stable parameter estimation.

Abstract

Performative predictions are forecasts which influence the outcomes they aim to predict, undermining the existence of correct forecasts and standard methods of elicitation and estimation. We show that conditioning forecasts on covariates that separate them from the outcome renders the target distribution forecast-invariant, guaranteeing well-posedness of the forecasting problem. However, even under this condition, classical proper scoring rules fail to elicit correct forecasts. We prove a general impossibility result and identify two solutions: (i) in decision-theoretic settings, elicitation of correct and incentive-compatible forecasts is possible if forecasts are separating; (ii) scoring with unbiased estimates of the divergence between the forecast and the induced distribution of the target variable yields correct forecasts. Applying these insights to parameter estimation, conditional forecasts and proper scoring rules enable performatively stable estimation of performatively correct parameters, resolving the issues raised by Perdomo et al. (2020). Our results expose fundamental limits of classical forecast evaluation and offer new tools for reliable and accurate forecasting in performative settings.