PersRM-R1: Enhance Personalized Reward Modeling with Reinforcement Learning

TL;DR

PersRM-R1 introduces a novel reinforcement learning framework that effectively captures personalized preferences from minimal data, improving alignment of language models with individual user values.

Contribution

It is the first reasoning-based reward modeling framework designed for personalized preferences using only a few exemplars, combining synthetic data and a two-stage training process.

Findings

Outperforms existing models of similar size in personalized accuracy

Matches larger models in generalization performance

Effective with limited personal data

Abstract

Reward models (RMs), which are central to existing post-training methods, aim to align LLM outputs with human values by providing feedback signals during fine-tuning. However, existing RMs struggle to capture nuanced, user-specific preferences, especially under limited data and across diverse domains. Thus, we introduce PersRM-R1, the first reasoning-based reward modeling framework specifically designed to identify and represent personal factors from only one or a few personal exemplars. To address challenges including limited data availability and the requirement for robust generalization, our approach combines synthetic data generation with a two-stage training pipeline consisting of supervised fine-tuning followed by reinforcement fine-tuning. Experimental results demonstrate that PersRM-R1 outperforms existing models of similar size and matches the performance of much larger models in both accuracy and generalizability, paving the way for more effective personalized LLMs.