A Critical Evaluation of AI Feedback for Aligning Large Language Models

TL;DR

This paper critically examines reinforcement learning with AI feedback (RLAIF) for large language models, revealing that simpler supervised fine-tuning with stronger teachers can outperform complex RL methods.

Contribution

It demonstrates that supervised fine-tuning with GPT-4 can surpass RLAIF, and provides insights into when RLAIF is beneficial versus when simpler methods suffice.

Findings

Supervised fine-tuning with GPT-4 outperforms RLAIF pipelines.

The benefits of RLAIF depend on model families and evaluation protocols.

RLAIF's improvements are largely due to the quality of the teacher model.

Abstract

Reinforcement learning with AI feedback (RLAIF) is a popular paradigm for improving the instruction-following abilities of powerful pre-trained language models. RLAIF first performs supervised fine-tuning (SFT) using demonstrations from a teacher model and then further fine-tunes the model with reinforcement learning (RL), using feedback from a critic model. While recent popular open-source models have demonstrated substantial improvements in performance from the RL step, in this paper we question whether the complexity of this RL step is truly warranted for AI feedback. We show that the improvements of the RL step are virtually entirely due to the widespread practice of using a weaker teacher model (e.g. GPT-3.5) for SFT data collection than the critic (e.g., GPT-4) used for AI feedback generation. Specifically, we show that simple supervised fine-tuning with GPT-4 as the teacher outperforms existing RLAIF pipelines. More generally, we find that the gains from RLAIF vary substantially across base model families, test-time evaluation protocols, and critic models. Finally, we provide a mechanistic explanation for when SFT may outperform the full two-step RLAIF pipeline as well as suggestions for making RLAIF maximally useful in practice.