Reasoning-Aware Proxy Reward Model using Process Mining

TL;DR

This paper introduces TACReward, a process mining-based reward model that evaluates intermediate reasoning steps in mathematical problem solving, improving the structural reasoning quality of language models without extra annotation.

Contribution

It proposes a novel reasoning-aware reward model that integrates process mining to assess stepwise reasoning, enhancing sparse reward reinforcement learning for mathematical tasks.

Findings

TACReward improves reasoning structure in language models.

Integration of TACReward leads to better performance on reasoning benchmarks.

The method does not require additional human annotations.

Abstract

Recent advances in sparse reward policy gradient methods have enabled effective reinforcement learning (RL) for language models post-training. However, for reasoning tasks such as mathematical problem solving, binarized outcome rewards provide limited feedback on intermediate reasoning steps. While some studies have attempted to address this issue by estimating overall reasoning quality, it remains unclear whether these rewards are reliable proxies for the quality of stepwise reasoning. In this study, we consider reasoning as a structured process and propose \textbf{TACReward}, the reward model that can be seamlessly integrated into sparse reward policy gradient methods without additional human annotation costs or architectural modifications. TACReward aggregates stepwise structural deviations between teacher and policy reasoning using process mining techniques, producing a scalar output reward range of $[0, 1]$ to indicate reasoning quality. Experiments on multiple mathematical reasoning benchmarks demonstrate that integrating the TACReward into sparse reward frameworks encourages the policy model to improve the structural quality of reasoning. Consequently, this leads to consistent performance improvements over existing sparse reward frameworks. Our code and model are publicly available at \href{https://github.com/Thrillcrazyer/TACReward}{GitHub} and \href{https://huggingface.co/Thrillcrazyer/TACReward7B}{HuggingFace}