Measuring the performance of sensors that report uncertainty

TL;DR

This paper introduces statistical scoring rules to validate and compare sensor outputs and inference algorithms that report uncertainty, demonstrated through cattle mass estimation from ground force data.

Contribution

It develops methods to evaluate sensor and algorithm accuracy using scoring rules, applicable to prediction intervals and parameter estimates with uncertainty.

Findings

Scoring rules effectively rank sensor accuracy and precision.

Application to cattle mass estimation identifies the most meaningful scoring methods.

Provides a framework for sensor validation using known true parameters.

Abstract

We provide methods to validate and compare sensor outputs, or inference algorithms applied to sensor data, by adapting statistical scoring rules. The reported output should either be in the form of a prediction interval or of a parameter estimate with corresponding uncertainty. Using knowledge of the `true' parameter values, scoring rules provide a method of ranking different sensors or algorithms for accuracy and precision. As an example, we apply the scoring rules to the inferred masses of cattle from ground force data and draw conclusions on which rules are most meaningful and in which way.