UltraFeedback: Boosting Language Models with Scaled AI Feedback

TL;DR

This paper introduces UltraFeedback, a large-scale AI feedback dataset generated automatically to improve open-source language models, addressing limitations of human feedback in size and diversity, and demonstrating its effectiveness in model alignment.

Contribution

The paper presents UltraFeedback, a novel large-scale, diversified AI feedback dataset that enhances language model alignment beyond human feedback limitations.

Findings

UltraFeedback contains over 1 million GPT-4 feedback instances.

Models trained with UltraFeedback outperform baselines on chat benchmarks.

AI feedback effectively improves open-source language model alignment.

Abstract

Learning from human feedback has become a pivot technique in aligning large language models (LLMs) with human preferences. However, acquiring vast and premium human feedback is bottlenecked by time, labor, and human capability, resulting in small sizes or limited topics of current datasets. This further hinders feedback learning as well as alignment research within the open-source community. To address this issue, we explore how to go beyond human feedback and collect high-quality \textit{AI feedback} automatically for a scalable alternative. Specifically, we identify \textbf{scale and diversity} as the key factors for feedback data to take effect. Accordingly, we first broaden instructions and responses in both amount and breadth to encompass a wider range of user-assistant interactions. Then, we meticulously apply a series of techniques to mitigate annotation biases for more reliable AI feedback. We finally present \textsc{UltraFeedback}, a large-scale, high-quality, and diversified AI feedback dataset, which contains over 1 million GPT-4 feedback for 250k user-assistant conversations from various aspects. Built upon \textsc{UltraFeedback}, we align a LLaMA-based model by best-of-$n$ sampling and reinforcement learning, demonstrating its exceptional performance on chat benchmarks. Our work validates the effectiveness of scaled AI feedback data in constructing strong open-source chat language models, serving as a solid foundation for future feedback learning research. Our data and models are available at https://github.com/thunlp/UltraFeedback.