How to show a probabilistic model is better

TL;DR

This paper introduces a straightforward theoretical framework based on proper scoring rules for comparing probabilistic models on real data, aiming to promote their broader use in machine learning performance evaluation.

Contribution

It presents a simple, practical approach grounded in established statistical theory for evaluating probabilistic models in machine learning.

Findings

Framework is easy to understand and verify.

Applicable to real-world data and models.

Encourages adoption of proper scoring rules in ML evaluation.

Abstract

We present a simple theoretical framework, and corresponding practical procedures, for comparing probabilistic models on real data in a traditional machine learning setting. This framework is based on the theory of proper scoring rules, but requires only basic algebra and probability theory to understand and verify. The theoretical concepts presented are well-studied, primarily in the statistics literature. The goal of this paper is to advocate their wider adoption for performance evaluation in empirical machine learning.