DC-W2S: Dual-Consensus Weak-to-Strong Training for Reliable Process Reward Modeling in Biological Reasoning

TL;DR

This paper introduces DC-W2S, a framework that improves the training of Process Reward Models in biological reasoning by selecting high-quality supervision signals from noisy data through dual consensus metrics, reducing reliance on expert annotations.

Contribution

The paper proposes a novel Dual-Consensus Weak-to-Strong framework that stratifies supervision signals using consensus metrics, enhancing the reliability of training PRMs with noisy supervision.

Findings

DC-W2S improves PRM robustness in biological reasoning tasks.

Strategic data curation outperforms large-scale noisy data training.

The framework reduces dependence on costly expert annotations.

Abstract

In scientific reasoning tasks, the veracity of the reasoning process is as critical as the final outcome. While Process Reward Models (PRMs) offer a solution to the coarse-grained supervision problems inherent in Outcome Reward Models (ORMs), their deployment is hindered by the prohibitive cost of obtaining expert-verified step-wise labels. This paper addresses the challenge of training reliable PRMs using abundant but noisy "weak" supervision. We argue that existing Weak-to-Strong Generalization (W2SG) theories lack prescriptive guidelines for selecting high-quality training signals from noisy data. To bridge this gap, we introduce the Dual-Consensus Weak-to-Strong (DC-W2S) framework. By intersecting Self-Consensus (SC) metrics among weak supervisors with Neighborhood-Consensus (NC) metrics in the embedding space, we stratify supervision signals into distinct reliability regimes. We then employ a curriculum of instance-level balanced sampling and label-level reliability-aware masking to guide the training process. We demonstrate that DC-W2S enables the training of robust PRMs for complex reasoning without exhaustive expert annotation, proving that strategic data curation is more effective than indiscriminate training on large-scale noisy datasets.