The Murphy Decomposition and the Calibration-Resolution Principle: A New Perspective on Forecast Evaluation

TL;DR

This paper introduces a unifying forecast evaluation framework based on calibration and resolution, generalizing existing principles and exposing limitations of current methods, with applications to economic forecasts.

Contribution

It presents a new theoretical perspective on forecast accuracy, extending the Murphy decomposition and the sharpness principle to all forecast types, and demonstrates its practical utility.

Findings

Reveals shortcomings of common forecast evaluation methods.

Provides new insights into economic forecast errors.

Highlights the potential of the decomposition approach.

Abstract

I provide a unifying perspective on forecast evaluation, characterizing accurate forecasts of all types, from simple point to complete probabilistic forecasts, in terms of two fundamental underlying properties, autocalibration and resolution, which can be interpreted as describing a lack of systematic mistakes and a high information content. This "calibration-resolution principle" gives a new insight into the nature of forecasting and generalizes the famous sharpness principle by Gneiting et al. (2007) from probabilistic to all types of forecasts. It amongst others exposes the shortcomings of several widely used forecast evaluation methods. The principle is based on a fully general version of the Murphy decomposition of loss functions, which I provide. Special cases of this decomposition are well-known and widely used in meteorology. Besides using the decomposition in this new theoretical way, after having introduced it and the underlying properties in a proper theoretical framework, accompanied by an illustrative example, I also employ it in its classical sense as a forecast evaluation method as the meteorologists do: As such, it unveils the driving forces behind forecast errors and complements classical forecast evaluation methods. I discuss estimation of the decomposition via kernel regression and then apply it to popular economic forecasts. Analysis of mean forecasts from the US Survey of Professional Forecasters and quantile forecasts derived from Bank of England fan charts indeed yield interesting new insights and highlight the potential of the method.