SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Margins

TL;DR

SeRA is a novel method that improves large language model alignment by using implicit reward margins for sample selection and preference bootstrapping, reducing overfitting and enhancing training efficiency.

Contribution

SeRA introduces a cost-effective approach combining implicit reward margins and preference bootstrapping to enhance DAA-based LLM alignment.

Findings

SeRA improves alignment accuracy on offline datasets.

SeRA reduces overfitting to spurious correlations.

SeRA demonstrates effectiveness across multiple instruction-following tasks.

Abstract

Direct alignment algorithms (DAAs), such as direct preference optimization (DPO), have become popular alternatives for Reinforcement Learning from Human Feedback (RLHF) due to their simplicity, efficiency, and stability. However, the preferences used in DAAs are usually collected before the alignment training begins and remain unchanged (off-policy). This can lead to two problems where the policy model (1) picks up on spurious correlations in the dataset (as opposed to learning the intended alignment expressed in the human preference labels), and (2) overfits to feedback on off-policy trajectories that have less likelihood of being generated by an updated policy model. To address these issues, we introduce Self-Reviewing and Alignment (SeRA), a cost-efficient and effective method that can be readily combined with existing DAAs. SeRA comprises of two components: (1) sample selection using implicit reward margins, which helps alleviate over-fitting to some undesired features, and (2) preference bootstrapping using implicit rewards to augment preference data with updated policy models in a cost-efficient manner. Extensive experimentation, including some on instruction-following tasks, demonstrate the effectiveness and generality of SeRA in training LLMs on offline preference datasets with DAAs.