Proper losses for discrete generative models

TL;DR

This paper introduces the concept of black-box proper losses for evaluating discrete generative models, characterizing their form and conditions for optimality using statistical estimation techniques.

Contribution

It provides a general construction and characterization of black-box proper losses, showing they must be polynomial and detailing sampling requirements, thus advancing evaluation methods for generative models.

Findings

Proper losses are polynomial in form.

Number of samples must exceed polynomial degree.

Cross-entropy-based loss cannot be black-box proper.

Abstract

We initiate the study of proper losses for evaluating generative models in the discrete setting. Unlike traditional proper losses, we treat both the generative model and the target distribution as black-boxes, only assuming ability to draw i.i.d. samples. We define a loss to be black-box proper if the generative distribution that minimizes expected loss is equal to the target distribution. Using techniques from statistical estimation theory, we give a general construction and characterization of black-box proper losses: they must take a polynomial form, and the number of draws from the model and target distribution must exceed the degree of the polynomial. The characterization rules out a loss whose expectation is the cross-entropy between the target distribution and the model. By extending the construction to arbitrary sampling schemes such as Poisson sampling, however, we show that one can construct such a loss.