Conformal Generative Modeling with Improved Sample Efficiency through Sequential Greedy Filtering

TL;DR

This paper introduces SCOPE-Gen, a sequential conformal prediction method for generative models that guarantees high-probability inclusion of valid examples, reducing costly admissibility evaluations in safety-critical applications.

Contribution

We propose a novel sequential conformal prediction approach that improves sample efficiency and provides rigorous guarantees for generative models in safety-critical contexts.

Findings

Significant reduction in admissibility evaluations compared to prior methods.

Effective control of prediction set admissibility via Markov chain factorization.

Successful application in natural language and molecular graph generation tasks.

Abstract

Generative models lack rigorous statistical guarantees for their outputs and are therefore unreliable in safety-critical applications. In this work, we propose Sequential Conformal Prediction for Generative Models (SCOPE-Gen), a sequential conformal prediction method producing prediction sets that satisfy a rigorous statistical guarantee called conformal admissibility control. This guarantee states that with high probability, the prediction sets contain at least one admissible (or valid) example. To this end, our method first samples an initial set of i.i.d. examples from a black box generative model. Then, this set is iteratively pruned via so-called greedy filters. As a consequence of the iterative generation procedure, admissibility of the final prediction set factorizes as a Markov chain. This factorization is crucial, because it allows to control each factor separately, using conformal prediction. In comparison to prior work, our method demonstrates a large reduction in the number of admissibility evaluations during calibration. This reduction is important in safety-critical applications, where these evaluations must be conducted manually by domain experts and are therefore costly and time consuming. We highlight the advantages of our method in terms of admissibility evaluations and cardinality of the prediction sets through experiments in natural language generation and molecular graph extension tasks.