Calibrated Multi-Level Quantile Forecasting

TL;DR

This paper introduces MultiQT, an online method that ensures calibrated, ordered, and robust multi-level quantile forecasts with no regret, improving forecast reliability in real-world applications like epidemics and energy.

Contribution

The paper presents MultiQT, a lightweight, adversarially robust, no-regret algorithm that guarantees calibration and order of multi-level quantile forecasts, adaptable to any forecaster.

Findings

MultiQT improves calibration of forecasts in epidemic and energy data.

The method maintains quantile order and does not degrade forecast accuracy.

Experimental results show significant calibration improvements with minimal loss.

Abstract

We develop an online method that guarantees calibration of quantile forecasts at multiple quantile levels simultaneously. In this work, a sequence of quantile forecasts is said to be calibrated provided that its $\alpha$-level predictions are greater than or equal to the target value at an $\alpha$ fraction of time steps, for each level $\alpha$. Our procedure, called the multi-level quantile tracker (MultiQT), is lightweight and wraps around any point or quantile forecaster to produce adjusted quantile forecasts that are guaranteed to be calibrated, even against adversarial distribution shifts. Critically, it does so while ensuring that the quantiles remain ordered, e.g., the 0.5-level quantile forecast will never be larger than the 0.6-level forecast. Moreover, the method has a no-regret guarantee, implying it will not degrade the performance of the existing forecaster (asymptotically), with respect to the quantile loss. In our experiments, we find that MultiQT significantly improves the calibration of real forecasters in epidemic and energy forecasting problems, while leaving the quantile loss largely unchanged or slightly improved.