Hummer: Towards Limited Competitive Preference Dataset

TL;DR

Hummer introduces a new preference dataset with reduced conflicting alignment objectives, leveraging AI feedback to improve alignment and robustness in language models.

Contribution

The paper presents Hummer, a novel preference dataset designed to minimize conflicts between alignment objectives, and develops reward models that effectively balance diverse alignment goals.

Findings

Hummer dataset reduces conflicts in preference data.

HummerRM reward models balance multiple alignment objectives.

Enhanced robustness against jailbreak attacks.

Abstract

Preference datasets are essential for incorporating human preferences into pre-trained language models, playing a key role in the success of Reinforcement Learning from Human Feedback. However, these datasets often demonstrate conflicting alignment objectives, leading to increased vulnerability to jailbreak attacks and challenges in adapting downstream tasks to prioritize specific alignment objectives without negatively impacting others. In this work, we introduce a novel statistical metric, Alignment Dimension Conflict, to quantify the degree of conflict within preference datasets. We then present \texttt{Hummer} and its fine-grained variant, \texttt{Hummer-F}, as innovative pairwise preference datasets with reduced-conflict alignment objectives. \texttt{Hummer} is built based on UltraFeedback and is enhanced by AI feedback from GPT-4, marking as the first preference dataset aimed at reducing the competition between alignment objectives. Furthermore, we develop reward models, HummerRM and HummerRM-F, which employ a hybrid sampling approach to balance diverse alignment objectives effectively. This sampling method positions HummerRM as an ideal model for domain-specific further fine-tuning and reducing vulnerabilities to attacks.