A New Perspective on Precision and Recall for Generative Models

TL;DR

This paper introduces a new framework for estimating Precision-Recall curves for generative models, providing a comprehensive statistical analysis and extending existing metrics.

Contribution

It proposes a novel binary classification-based method for full PR curve estimation, with theoretical bounds and extensions of existing PR metrics.

Findings

Derived a minimax upper bound on PR estimation risk.

Extended landmark PR metrics to entire curves.

Analyzed PR curve behaviors across various experimental settings.

Abstract

With the recent success of generative models in image and text, the question of their evaluation has recently gained a lot of attention. While most methods from the state of the art rely on scalar metrics, the introduction of Precision and Recall (PR) for generative model has opened up a new avenue of research. The associated PR curve allows for a richer analysis, but their estimation poses several challenges. In this paper, we present a new framework for estimating entire PR curves based on a binary classification standpoint. We conduct a thorough statistical analysis of the proposed estimates. As a byproduct, we obtain a minimax upper bound on the PR estimation risk. We also show that our framework extends several landmark PR metrics of the literature which by design are restrained to the extreme values of the curve. Finally, we study the different behaviors of the curves obtained experimentally in various settings.