Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models

TL;DR

This paper introduces verifiable process supervision (VPS), a post-training framework that enhances language models' reasoning quality without sacrificing accuracy, demonstrated on a chess domain with deterministic verification.

Contribution

VPS jointly optimizes prediction accuracy and reasoning quality using structured reasoning induction, process-level rewards, and adaptive weighting, improving reasoning reliability.

Findings

VPS preserves accuracy while significantly improving reasoning quality.

Accuracy-only RL degrades reasoning, increasing errors and reducing consistency.

VPS outperforms traditional RL in verifiable reasoning tasks, restoring internal consistency.

Abstract

Training language models to produce both correct answers and sound reasoning remains an open challenge. Reinforcement learning with verifiable rewards typically optimizes only final outcomes, which can lead to a failure mode where task accuracy improves while reasoning becomes less accurate, less complete, or even internally inconsistent. We propose verifiable process supervision (VPS), a post-training framework for verifiable domains that jointly optimizes prediction accuracy and reasoning quality. We first apply supervised fine-tuning to induce a structured reasoning format, enabling syntactic extraction of intermediate claims that are evaluated against ground-truth signals to form process-level rewards. To address the heterogeneous difficulty of reasoning subtasks, we introduce adaptive reward weighting that prioritizes components with the largest remaining errors, creating an implicit curriculum. We evaluate VPS on chess, a controlled testbed where reasoning steps can be deterministically verified against engine signals. While accuracy-only RL improves move accuracy, it sharply degrades reasoning quality, increasing win-rate error by up to 112% and reducing internal consistency by up to 69%. In contrast, VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation. At matched accuracy, judge evaluation also prefers the process-supervised models. A reasoning-space analysis further shows that, without a structured prior, accuracy-only RL converges to budget-dependent shortcuts rather than sound multi-step reasoning. These results show that VPS enables language models to reason both accurately and reliably in verifiable domains.