Scoring Interval Forecasts: Equal-Tailed, Shortest, and Modal Interval

TL;DR

This paper investigates the elicitability of various predictive intervals, identifying which are uniquely minimizable by scoring functions, and provides guidance for evaluating interval forecasts.

Contribution

It characterizes the elicitability of equal-tailed, modal, and shortest intervals, clarifying their suitability for forecast evaluation.

Findings

Equal-tailed and modal intervals are elicitable with specific scoring functions.

Shortest interval is not elicitable under practical distribution classes.

Guides the choice of performance measures in forecast competitions.

Abstract

We consider different types of predictive intervals and ask whether they are elicitable, i.e. are unique minimizers of a loss or scoring function in expectation. The equal-tailed interval is elicitable, with a rich class of suitable loss functions, though subject to translation invariance, or positive homogeneity and differentiability, the Winkler interval score becomes a unique choice. The modal interval also is elicitable, with a sole consistent scoring function, up to equivalence. However, the shortest interval fails to be elicitable relative to practically relevant classes of distributions. These results provide guidance in interval forecast evaluation and support recent choices of performance measures in forecast competitions.