Think Twice: Branch-and-Rethink Reasoning Reward Model

TL;DR

This paper introduces branch-and-rethink reward modeling, a two-turn process that enhances reasoning accuracy by focusing on critical dimensions and re-evaluating hypotheses, leading to improved performance on benchmark tasks.

Contribution

The paper proposes a novel two-turn reward model that mimics think-twice reasoning, reducing judgment diffusion and improving sensitivity to errors in reward evaluation.

Findings

Achieves state-of-the-art results on three reward modeling benchmarks.

Reduces judgment diffusion by focusing on critical evaluation dimensions.

Enhances sensitivity to subtle errors through targeted rethinking.

Abstract

Large language models (LLMs) increasingly rely on thinking models that externalize intermediate steps and allocate extra test-time compute, with think-twice strategies showing that a deliberate second pass can elicit stronger reasoning. In contrast, most reward models (RMs) still compress many quality dimensions into a single scalar in one shot, a design that induces judgment diffusion: attention spreads across evaluation criteria, yielding diluted focus and shallow analysis. We introduce branch-and-rethink (BR-RM), a two-turn RM that transfers the think-twice principle to reward modeling. Turn 1 performs adaptive branching, selecting a small set of instance-critical dimensions (such as factuality and safety) and sketching concise, evidence-seeking hypotheses. Turn 2 executes branch-conditioned rethinking, a targeted reread that tests those hypotheses and scrutinizes only what matters most. We train with GRPO-style reinforcement learning over structured two-turn traces using a simple binary outcome reward with strict format checks, making the approach compatible with standard RLHF pipelines. By converting all-at-once scoring into focused, second-look reasoning, BR-RM reduces judgment diffusion and improves sensitivity to subtle yet consequential errors while remaining practical and scalable. Experimental results demonstrate that our model achieves state-of-the-art performance on three challenging reward modeling benchmarks across diverse domains.