Let's Reinforce Step by Step
Sarah Pan, Vladislav Lialin, Sherin Muckatira, and Anna Rumshisky

TL;DR
This paper investigates reinforcement learning from human feedback to improve language models' reasoning, comparing outcome and process-supervised reward models, and highlights the importance of reward aggregation functions.
Contribution
It introduces and evaluates process-supervised reward models for reasoning tasks, revealing their strengths and limitations compared to outcome-based rewards.
Findings
PRM improves accuracy on simple math tasks
PRM reduces performance on complex reasoning tasks
Reward aggregation functions significantly impact model performance
Abstract
While recent advances have boosted LM proficiency in linguistic benchmarks, LMs consistently struggle to reason correctly on complex tasks like mathematics. We turn to Reinforcement Learning from Human Feedback (RLHF) as a method with which to shape model reasoning processes. In particular, we explore two reward schemes, outcome-supervised reward models (ORMs) and process-supervised reward models (PRMs), to optimize for logical reasoning. Our results show that the fine-grained reward provided by PRM-based methods enhances accuracy on simple mathematical reasoning (GSM8K) while, unexpectedly, reducing performance in complex tasks (MATH). Furthermore, we show the critical role reward aggregation functions play in model performance. Providing promising avenues for future research, our study underscores the need for further exploration into fine-grained reward modeling for more reliable…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques
