Coarse-to-Fine Process Reward Modeling for Mathematical Reasoning

TL;DR

This paper introduces CFPRM, a hierarchical coarse-to-fine approach for process reward modeling in mathematical reasoning, reducing redundancy in LLM-generated reasoning steps and improving training data quality.

Contribution

We propose a novel hierarchical coarse-to-fine strategy for process reward modeling that effectively reduces redundancy and enhances reasoning data quality.

Findings

CFPRM improves reasoning accuracy across datasets.

Hierarchical refinement captures essential reasoning steps.

Method demonstrates versatility across multiple loss criteria.

Abstract

The Process Reward Model (PRM) plays a crucial role in mathematical reasoning tasks, requiring high-quality supervised process data. However, we observe that reasoning steps generated by Large Language Models (LLMs) often fail to exhibit strictly incremental information, leading to redundancy that can hinder effective reasoning. To address this issue, we propose CFPRM, a simple yet effective coarse-to-fine strategy. Instead of focusing on the detection of redundant steps, our approach first establishes a coarse-grained window to merge adjacent reasoning steps into unified, holistic steps. The window size is then progressively reduced to extract fine-grained reasoning steps, enabling data collection at multiple granularities for training. By leveraging this hierarchical refinement process, CFPRM mitigates redundancy while preserving essential fine-grained knowledge. Extensive experiments on two reasoning datasets across three loss criteria validate the CFPRM's effectiveness and versatility.