Does RLHF Scale? Exploring the Impacts From Data, Model, and Method

TL;DR

This paper investigates how various factors like data, model size, and inference budget influence the scalability of RLHF in large language models, revealing diminishing returns and proposing optimization strategies.

Contribution

It systematically analyzes the impacts of data diversity, model size, and inference on RLHF performance, providing insights into its scaling limitations and optimization strategies.

Findings

Increasing data diversity improves reward model performance.

More response samples initially boost policy performance but plateau quickly.

Larger reward models offer modest gains, with diminishing returns in scaling.

Abstract

This study explores the scaling properties of Reinforcement Learning from Human Feedback (RLHF) in Large Language Models (LLMs). Although RLHF is considered an important step in post-training of LLMs, its scaling potential is still largely unknown. We systematically analyze key components in the RLHF framework--model size, data composition, and inference budget--and their impacts on performance. Our findings show that increasing data diversity and volume improves reward model performance, helping process-supervision models scale better. For policy training, more response samples per prompt boost performance initially but quickly plateau. And larger reward models offer modest gains in policy training. In addition, larger policy models benefit less from RLHF with a fixed reward model. Overall, RLHF scales less efficiently than pretraining, with diminishing returns from additional computational resources. Based on these observations, we propose strategies to optimize RLHF performance within computational limits.