Are You Sure? Rank Them Again: Repeated Ranking For Better Preference Datasets

TL;DR

This paper introduces Repeat Ranking, a method that improves large language model training by re-evaluating responses multiple times and training only on consistently ranked responses, leading to better alignment with human preferences.

Contribution

The paper proposes Repeat Ranking, a novel approach that enhances dataset quality for RLAIF by focusing on consistently ranked responses, outperforming standard practices.

Findings

Outperforms standard training on all prompts in multilingual benchmarks

Highlights the importance of response consistency in dataset quality

Demonstrates a trade-off between data quality and quantity in RLAIF

Abstract

Training Large Language Models (LLMs) with Reinforcement Learning from AI Feedback (RLAIF) aligns model outputs more closely with human preferences. This involves an evaluator model ranking multiple candidate responses to user prompts. However, the rankings from popular evaluator models such as GPT-4 can be inconsistent. We propose the Repeat Ranking method - where we evaluate the same responses multiple times and train only on those responses which are consistently ranked. Using 2,714 prompts in 62 languages, we generated responses from 7 top multilingual LLMs and had GPT-4 rank them five times each. Evaluating on MT-Bench chat benchmarks in six languages, our method outperformed the standard practice of training on all available prompts. Our work highlights the quality versus quantity trade-off in RLAIF dataset generation and offers a stackable strategy for enhancing dataset and thus model quality.