Ranking probabilistic forecasting models with different loss functions

TL;DR

This paper proposes new statistical metrics based on pinball loss and empirical coverage to rank probabilistic forecasting models, demonstrating that coverage-based ranking aligns better with economic performance in trading strategies.

Contribution

It introduces novel performance metrics for probabilistic forecast ranking and evaluates their effectiveness in predicting trading profitability.

Findings

Coverage-based ranking correlates with higher trading profits.

Proposed metrics outperform traditional ranking methods.

Quantile coverage is a key indicator of forecast usefulness.

Abstract

In this study, we introduced various statistical performance metrics, based on the pinball loss and the empirical coverage, for the ranking of probabilistic forecasting models. We tested the ability of the proposed metrics to determine the top performing forecasting model and investigated the use of which metric corresponds to the highest average per-trade profit in the out-of-sample period. Our findings show that for the considered trading strategy, ranking the forecasting models according to the coverage of quantile forecasts used in the trading hours exhibits a superior economic performance.