Toward a Characterization of Loss Functions for Distribution Learning

TL;DR

This paper investigates the properties of loss functions for distribution learning, proposing criteria for good losses, and showing that with certain restrictions, multiple losses including log loss are suitable, encouraging domain-specific choices.

Contribution

It introduces axiomatic criteria for evaluating loss functions in distribution learning and demonstrates that, under distribution restrictions, various losses meet these criteria.

Findings

No loss function satisfies all criteria without restrictions.

Restricting to calibrated distributions allows multiple losses to be suitable.

Encourages domain-specific loss function selection in distribution learning.

Abstract

In this work we study loss functions for learning and evaluating probability distributions over large discrete domains. Unlike classification or regression where a wide variety of loss functions are used, in the distribution learning and density estimation literature, very few losses outside the dominant $log\ loss$ are applied. We aim to understand this fact, taking an axiomatic approach to the design of loss functions for learning distributions. We start by proposing a set of desirable criteria that any good loss function should satisfy. Intuitively, these criteria require that the loss function faithfully evaluates a candidate distribution, both in expectation and when estimated on a few samples. Interestingly, we observe that \emph{no loss function} possesses all of these criteria. However, one can circumvent this issue by introducing a natural restriction on the set of candidate distributions. Specifically, we require that candidates are $calibrated$ with respect to the target distribution, i.e., they may contain less information than the target but otherwise do not significantly distort the truth. We show that, after restricting to this set of distributions, the log loss, along with a large variety of other losses satisfy the desired criteria. These results pave the way for future investigations of distribution learning that look beyond the log loss, choosing a loss function based on application or domain need.