Loss Functions for Measuring the Accuracy of Nonnegative Cross-Sectional Predictions

TL;DR

This paper develops an axiomatic loss function framework for measuring the accuracy of nonnegative cross-sectional predictions, incorporating decision-maker preferences and enabling optimal weighting and bias assessment.

Contribution

It introduces a novel, axiomatic loss function approach for evaluating predictions, including resource allocations, with methods for parameter estimation and bias measurement.

Findings

Loss functions can be estimated via linear regression.

The framework extends to weighted averages for optimal predictions.

Comparison with existing loss functions shows advantages.

Abstract

Measuring the accuracy of cross-sectional predictions is a subjective problem. Generally, this problem is avoided. In contrast, this paper confronts subjectivity up front by eliciting an impartial decision-maker's preferences. These preferences are embedded into an axiomatically-derived loss function, one of the simplest version of which is described. The parameters of the loss function can be estimated by linear regression. Specification tests for this function are described. This framework is extended to weighted averages of estimates to find the optimal weightings. A special case occurs when the predictions represent resource allocations: the apportionment literature is used to construct the Webster-Saint Lag\"ue Rule, a particular parametrization of the loss function. These loss functions are compared to those existing in the literature. Finally, a family of bias measures are created using signed versions of these loss functions.