PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling

TL;DR

PaTaRM introduces a novel reward modeling approach that leverages pairwise data for pointwise training and dynamically adapts evaluation criteria, significantly improving language model alignment with human preferences.

Contribution

It presents a new method combining pairwise and pointwise signals through a preference-aware mechanism and task-adaptive rubric, enhancing reward model training and evaluation.

Findings

Achieves 8.7% average improvement on RewardBench and RMBench.

Boosts downstream RLHF performance by 13.6% on IFEval and InFoBench.

Enables robust pointwise training without explicit rating labels.

Abstract

Reward models (RMs) are central to reinforcement learning from human feedback (RLHF), providing the critical supervision signals that align large language models (LLMs) with human preferences. Generative reward models (GRMs) provide greater interpretability than traditional scalar RMs, but they come with a critical trade-off: pairwise methods are hindered by a training-inference mismatch, while pointwise methods require expensive absolute annotations. To bridge this gap, we propose the Preference-aware Task-adaptive Reward Model (PaTaRM). Unlike prior approaches, PaTaRM enables robust pointwise training using readily available pairwise data via a novel Preference-Aware Reward (PAR) mechanism, eliminating the need for explicit rating labels. Furthermore, it incorporates a Task-Adaptive Rubric system that dynamically generates instance-specific criteria for precise evaluation. Extensive experiments demonstrate that PATRM achieves a 8.7% average improvement on RewardBench and RMBench across Qwen3-8B/14B models. Crucially, it boosts downstream RLHF performance by an average relative improvement of 13.6% across IFEval and InFoBench, validating its effectiveness for policy alignment. Our code is available at https://github.com/JaneEyre0530/PaTaRM.