PRISM: A Unified Framework for Post-Training LLMs Without Verifiable Rewards

TL;DR

PRISM introduces a unified post-training framework for LLMs that combines a Process Reward Model with internal confidence metrics to improve training stability and performance without relying on external labels or human supervision.

Contribution

The paper proposes PRISM, a novel framework that effectively integrates a Process Reward Model with internal confidence to guide LLM training without ground-truth labels.

Findings

PRISM achieves more stable training compared to existing methods.

Combining PRM with self-certainty improves test performance.

PRISM reduces reliance on costly human supervision.

Abstract

Current techniques for post-training Large Language Models (LLMs) rely either on costly human supervision or on external verifiers to boost performance on tasks such as mathematical reasoning and code generation. However, as LLMs improve their problem-solving, any further improvement will potentially require high-quality solutions to difficult problems that are not available to humans. As a result, learning from unlabeled data is becoming increasingly attractive in the research community. Existing methods extract learning signal from a model's consistency, either by majority voting or by converting the model's internal confidence into reward. Although internal consistency metric such as entropy or self-certainty require no human intervention, as we show in this work, these are unreliable signals for large-scale and long-term training. To address the unreliability, we propose PRISM, a unified training framework that uses a Process Reward Model (PRM) to guide learning alongside model's internal confidence in the absence of ground-truth labels. We show that effectively combining PRM with self-certainty can lead to both stable training and better test-time performance, and also keep the model's internal confidence in check. Code available at https://github.com/ghimiremukesh/PRISM.