Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective Ambiguity

TL;DR

This paper introduces Act-Adaptive Margin (AAM), a method that dynamically calibrates reward models to better handle subjective ambiguity, significantly improving performance in subjective reward modeling and alignment tasks.

Contribution

AAM provides a novel way to calibrate preference margins dynamically, enhancing reward models' ability to handle subjective ambiguity without extra human annotation.

Findings

AAM improves Bradley-Terry reward models by 2.95% in general tasks.

AAM enhances performance by 4.85% in subjective role-playing tasks.

Reward models with AAM achieve state-of-the-art results on CharacterEval and Charm.

Abstract

Currently, most reinforcement learning tasks focus on domains like mathematics and programming, where verification is relatively straightforward. However, in subjective tasks such as role-playing, alignment techniques struggle to make progress, primarily because subjective reward modeling using the Bradley-Terry model faces significant challenges when dealing with ambiguous preferences. To improve reward modeling in subjective tasks, this paper proposes AAM (\textbf{\underline{A}}ct-\textbf{\underline{A}}daptive \textbf{\underline{M}}argin), which enhances reward modeling by dynamically calibrating preference margins using the model's internal parameter knowledge. We design two versions of AAM that efficiently generate contextually-appropriate preference gaps without additional human annotation. This approach fundamentally improves how reward models handle subjective rewards by better integrating generative understanding with preference scoring. To validate AAM's effectiveness in subjective reward modeling, we conduct evaluations on RewardBench, JudgeBench, and challenging role-playing tasks. Results show that AAM significantly improves subjective reward modeling performance, enhancing Bradley-Terry reward models by 2.95\% in general tasks and 4.85\% in subjective role-playing tasks. Furthermore, reward models trained with AAM can help downstream alignment tasks achieve better results. Our test results show that applying rewards generated by AAM-Augmented RM to preference learning techniques (e.g., GRPO) achieves state-of-the-art results on CharacterEval and Charm. Code and dataset are available at https://github.com/calubkk/AAM.