Diverse Preference Learning for Capabilities and Alignment

TL;DR

This paper introduces Soft Preference Learning, a novel method that enhances diversity and societal viewpoint representation in LLM outputs by decoupling entropy and cross-entropy in preference learning.

Contribution

It proposes a new preference learning approach that improves diversity and societal perspective representation in LLMs, addressing limitations of existing alignment algorithms.

Findings

LLMs trained with Soft Preference Learning show higher accuracy on complex tasks.

Outputs exhibit greater semantic and lexical diversity.

Models better represent diverse societal viewpoints and have improved logit calibration.

Abstract

The ability of LLMs to represent diverse perspectives is critical as they increasingly impact society. However, recent studies reveal that alignment algorithms such as RLHF and DPO significantly reduce the diversity of LLM outputs. Not only do aligned LLMs generate text with repetitive structure and word choice, they also approach problems in more uniform ways, and their responses reflect a narrower range of societal perspectives. We attribute this problem to the KL divergence regularizer employed in preference learning algorithms. This causes the model to systematically overweight majority opinions and sacrifice diversity in its outputs. To address this, we propose Soft Preference Learning, which decouples the entropy and cross-entropy terms in the KL penalty - allowing for fine-grained control over LLM generation diversity. From a capabilities perspective, LLMs trained using Soft Preference Learning attain higher accuracy on difficult repeated sampling tasks and produce outputs with greater semantic and lexical diversity. From an alignment perspective, they are capable of representing a wider range of societal viewpoints and display improved logit calibration. Notably, Soft Preference Learning resembles, but is a Pareto improvement over, standard temperature scaling.