Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimization

TL;DR

This paper introduces a novel online RLHF method that uses only prompts for self-rewarding, reducing reliance on judgment models, and improves model alignment by generating challenging negatives to better capture human preferences.

Contribution

The paper proposes an only-prompting self-rewarding online algorithm that generates preference data without judgment models and employs fine-grained control over training difficulty.

Findings

Achieved 34.5% win rate on AlpacaEval 2.0

Significantly improved performance of base models

Demonstrated effectiveness on Mistral-7B and Mistral-Instruct-7B

Abstract

We address the challenge of online Reinforcement Learning from Human Feedback (RLHF) with a focus on self-rewarding alignment methods. In online RLHF, obtaining feedback requires interaction with the environment, which can be costly when using additional reward models or the GPT-4 API. Current self-rewarding approaches rely heavily on the discriminator's judgment capabilities, which are effective for large-scale models but challenging to transfer to smaller ones. To address these limitations, we propose a novel, only-prompting self-rewarding online algorithm that generates preference datasets without relying on judgment capabilities. Additionally, we employ fine-grained arithmetic control over the optimality gap between positive and negative examples, generating more hard negatives in the later stages of training to help the model better capture subtle human preferences. Finally, we conduct extensive experiments on two base models, Mistral-7B and Mistral-Instruct-7B, which significantly bootstrap the performance of the reference model, achieving 34.5% in the Length-controlled Win Rates of AlpacaEval 2.0.