WorldPM: Scaling Human Preference Modeling

TL;DR

This paper introduces WorldPM, a scalable human preference modeling framework that leverages large-scale data and models to improve preference alignment, showing significant gains across multiple benchmarks and evaluation metrics.

Contribution

We propose WorldPM, a unified preference modeling approach demonstrating scalable improvements in preference prediction and fine-tuning for large language models.

Findings

Preference metrics scale with data and model size, especially adversarial and objective metrics.

WorldPM improves performance on 7 benchmarks with over 5% gains.

Integration into RLHF pipeline yields 4-8% improvements in evaluations.

Abstract

Motivated by scaling laws in language modeling that demonstrate how test loss scales as a power law with model and dataset sizes, we find that similar laws exist in preference modeling. We propose World Preference Modeling$ (WorldPM) to emphasize this scaling potential, where World Preference embodies a unified representation of human preferences. In this paper, we collect preference data from public forums covering diverse user communities, and conduct extensive training using 15M-scale data across models ranging from 1.5B to 72B parameters. We observe distinct patterns across different evaluation metrics: (1) Adversarial metrics (ability to identify deceptive features) consistently scale up with increased training data and base model size; (2) Objective metrics (objective knowledge with well-defined answers) show emergent behavior in larger language models, highlighting WorldPM's scalability potential; (3) Subjective metrics (subjective preferences from a limited number of humans or AI) do not demonstrate scaling trends. Further experiments validate the effectiveness of WorldPM as a foundation for preference fine-tuning. Through evaluations on 7 benchmarks with 20 subtasks, we find that WorldPM broadly improves the generalization performance across human preference datasets of varying sizes (7K, 100K and 800K samples), with performance gains exceeding 5% on many key subtasks. Integrating WorldPM into our internal RLHF pipeline, we observe significant improvements on both in-house and public evaluation sets, with notable gains of 4% to 8% in our in-house evaluations.