Sensitivity Measures Based on Scoring Functions

TL;DR

This paper introduces a comprehensive framework for constructing and analyzing sensitivity measures based on scoring functions, applicable to various elicitable functionals, and demonstrates their properties and applications through examples and simulations.

Contribution

It develops a unified approach to sensitivity analysis using scoring functions for elicitable functionals, including new properties and visualization tools like Murphy diagrams.

Findings

Score-based sensitivities quantify the value of information in predictions.

The framework applies to mean, Value-at-Risk, and Expected Shortfall functionals.

Simulation shows effective estimation of sensitivities in complex models.

Abstract

We propose a holistic framework for constructing sensitivity measures for any elicitable functional $T$ of a response variable. The sensitivity measures, termed score-based sensitivities, are constructed via scoring functions that are (strictly) consistent for $T$. These score-based sensitivities quantify the relative improvement in predictive accuracy when available information, e.g., from explanatory variables, is used ideally. We establish intuitive and desirable properties of these sensitivities and discuss advantageous choices of scoring functions leading to scale-invariant sensitivities. Since elicitable functionals typically possess rich classes of (strictly) consistent scoring functions, we demonstrate how Murphy diagrams can provide a picture of all score-based sensitivity measures. We discuss the family of score-based sensitivities for the mean functional (of which the Sobol indices are a special case) and risk functionals such as Value-at-Risk, and the pair Value-at-Risk and Expected Shortfall. The sensitivity measures are illustrated using numerous examples, including the Ishigami--Homma test function. In a simulation study, estimation of score-based sensitivities for a non-linear insurance portfolio is performed using neural nets.