Fine-Tuning Language Models from Human Preferences

TL;DR

This paper demonstrates how reward learning from human preferences can fine-tune language models for tasks like sentiment continuation and summarization, improving alignment with human judgments.

Contribution

It applies reward learning to natural language tasks using human comparisons, advancing RL safety and practicality in language model fine-tuning.

Findings

Good results with only 5,000 human comparisons for stylistic continuation

Models trained with 60,000 comparisons produce summaries that align well with human preferences

Models can exploit simple heuristics, highlighting challenges in reward modeling

Abstract

Reward learning enables the application of reinforcement learning (RL) to tasks where reward is defined by human judgment, building a model of reward by asking humans questions. Most work on reward learning has used simulated environments, but complex information about values is often expressed in natural language, and we believe reward learning for language is a key to making RL practical and safe for real-world tasks. In this paper, we build on advances in generative pretraining of language models to apply reward learning to four natural language tasks: continuing text with positive sentiment or physically descriptive language, and summarization tasks on the TL;DR and CNN/Daily Mail datasets. For stylistic continuation we achieve good results with only 5,000 comparisons evaluated by humans. For summarization, models trained with 60,000 comparisons copy whole sentences from the input but skip irrelevant preamble; this leads to reasonable ROUGE scores and very good performance according to our human labelers, but may be exploiting the fact that labelers rely on simple heuristics.