Models That Prove Their Own Correctness

TL;DR

This paper introduces Self-Proving models that can verify the correctness of their outputs with high probability using an interactive proof system, providing stronger guarantees than traditional accuracy metrics.

Contribution

It proposes a theoretically-founded framework for training models that can prove their output correctness via an interactive verifier, a novel approach to model reliability.

Findings

Self-Proving models can reliably verify most of their outputs.

Two generic training methods for Self-Proving models are introduced: Transcript Learning and Reinforcement Learning from Verifier Feedback.

The approach guarantees that incorrect outputs are detected by the verifier, enhancing trustworthiness.

Abstract

How can we trust the correctness of a learned model on a particular input of interest? Model accuracy is typically measured on average over a distribution of inputs, giving no guarantee for any fixed input. This paper proposes a theoretically-founded solution to this problem: to train Self-Proving models that prove the correctness of their output to a verification algorithm $V$ via an Interactive Proof. Self-Proving models satisfy that, with high probability over an input sampled from a given distribution, the model generates a correct output and successfully proves its correctness to $V$. The soundness property of $V$ guarantees that, for every input, no model can convince $V$ of the correctness of an incorrect output. Thus, a Self-Proving model proves correctness of most of its outputs, while all incorrect outputs (of any model) are detected by $V$. We devise and analyze two generic methods for learning Self-Proving models: Transcript Learning (TL) which relies on access to transcripts of accepting interactions, and Reinforcement Learning from Verifier Feedback (RLVF) which trains a model by emulating interactions with the verifier.