Score-based calibration testing for multivariate forecast distributions

TL;DR

This paper introduces two novel calibration tests for multivariate forecast distributions based on proper scoring rules, addressing limitations of existing methods and demonstrating strong performance in simulations and real data applications.

Contribution

It proposes a general framework for multivariate calibration testing and develops new tests that outperform existing methods in size and power.

Findings

New tests have good size and power properties in simulations.

The tests effectively identify calibration issues in macroeconomic and financial forecasts.

Proposed framework overcomes challenges faced by PIT-based tests in multivariate settings.

Abstract

Calibration tests based on the probability integral transform (PIT) are routinely used to assess the quality of univariate distributional forecasts. However, PIT-based calibration tests for multivariate distributional forecasts face various challenges. We propose two new types of tests based on proper scoring rules, which overcome these challenges. They arise from a general framework for calibration testing in the multivariate case, introduced in this work. The new tests have good size and power properties in simulations and solve various problems of existing tests. We apply the tests to forecast distributions for macroeconomic and financial time series data.