Step-wise Rubric Rewards for LLM Reasoning

TL;DR

This paper introduces Step-wise Rubrics as Rewards (SRaR), a novel reinforcement learning framework that improves reasoning accuracy in large language models by providing fine-grained, step-level supervision and addressing reward hacking issues.

Contribution

SRaR uses an LLM judge for step attribution, normalizes per-step scores, and combines rewards with a decoupled advantage estimator, significantly enhancing reasoning performance.

Findings

SRaR improves accuracy by 3.57 and 2.75 points on two benchmarks.

Raises Faithful Reasoning Rate from 34.5% to 46.7%.

Reduces self-correction looping from 48.1% to 26.5%.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in large language models, but rewards only final-answer correctness with no supervision over intermediate steps. Rubric-based methods such as Rubrics as Rewards (RaR) introduce finer-grained supervision by scoring rollouts against structured criteria, yet the rubric scores are still aggregated into a single scalar applied to the entire response, causing three weaknesses: loss of multi-criterion structure, uniform supervision of correct and incorrect steps, and reward hacking through unbounded self-correction. On 1,000 problems, we find 18.2% of steps in correct-answer responses are wrong yet positively rewarded, while 49.9% of steps in incorrect-answer responses are correct yet penalized. We introduce Step-wise Rubrics as Rewards (SRaR), an RLVR framework that (i) uses an LLM judge to attribute each rubric item to a specific reasoning step, (ii) normalizes per-step rubric scores across rollouts so only steps whose quality varies produce a learning signal, and (iii) combines the per-step reward with the outcome reward through a decoupled advantage estimator that keeps the outcome baseline stable. We further build a 16K-problem rubric dataset by contrastively distilling rubric items from correct and flawed reasoning paths sampled from a strong model. Across six mathematical reasoning benchmarks, SRaR improves average accuracy over RaR by 3.57 points on Qwen3-8B and 2.75 points on Qwen3-32B, raises the Faithful Reasoning Rate on AIME 2025 from 34.5% to 46.7%, and reduces self-correction looping from 48.1% to 26.5%.