P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling

TL;DR

P-GenRM introduces a novel personalized reward model that uses test-time user-based scaling and clustering to improve alignment of language models with individual user preferences, achieving state-of-the-art results.

Contribution

It proposes P-GenRM, a personalized generative reward model with test-time user-based scaling, enabling better generalization and adaptation to individual user preferences.

Findings

Achieves 2.31% improvement on personalized reward benchmarks.

Demonstrates strong generalization on out-of-distribution data.

Test-time user-based scaling adds an extra 3% performance boost.

Abstract

Personalized alignment of large language models seeks to adapt responses to individual user preferences, typically via reinforcement learning. A key challenge is obtaining accurate, user-specific reward signals in open-ended scenarios. Existing personalized reward models face two persistent limitations: (1) oversimplifying diverse, scenario-specific preferences into a small, fixed set of evaluation principles, and (2) struggling with generalization to new users with limited feedback. To this end, we propose P-GenRM, the first Personalized Generative Reward Model with test-time user-based scaling. P-GenRM transforms preference signals into structured evaluation chains that derive adaptive personas and scoring rubrics across various scenarios. It further clusters users into User Prototypes and introduces a dual-granularity scaling mechanism: at the individual level, it adaptively scales and aggregates each user's scoring scheme; at the prototype level, it incorporates preferences from similar users. This design mitigates noise in inferred preferences and enhances generalization to unseen users through prototype-based transfer. Empirical results show that P-GenRM achieves state-of-the-art results on widely-used personalized reward model benchmarks, with an average improvement of 2.31%, and demonstrates strong generalization on an out-of-distribution dataset. Notably, Test-time User-based scaling provides an additional 3% boost, demonstrating stronger personalized alignment with test-time scalability.