As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss

TL;DR

This paper introduces a simple, hyper-parameter-free bidirectional negative feedback loss for aligning large language models, improving stability and performance on reasoning tasks while maintaining efficiency.

Contribution

It proposes a novel BNF loss that simplifies LLM alignment by removing the need for pairwise data and hyper-parameter tuning, enhancing stability and reasoning ability.

Findings

BNF achieves comparable QA performance to state-of-the-art methods.

BNF shows significantly less performance decline on reasoning benchmarks.

Extensive experiments validate BNF's effectiveness and stability.

Abstract

Direct Preference Optimization (DPO) has emerged as a more computationally efficient alternative to Reinforcement Learning from Human Feedback (RLHF) with Proximal Policy Optimization (PPO), eliminating the need for reward models and online sampling. Despite these benefits, DPO and its variants remain sensitive to hyper-parameters and prone to instability, particularly on mathematical datasets. We argue that these issues arise from the unidirectional likelihood-derivative negative feedback inherent in the log-likelihood loss function. To address this, we propose a novel LLM alignment loss that establishes a stable Bidirectional Negative Feedback (BNF) during optimization. Our proposed BNF loss eliminates the need for pairwise contrastive losses and does not require any extra tunable hyper-parameters or pairwise preference data, streamlining the alignment pipeline to be as simple as supervised fine-tuning. We conduct extensive experiments across two challenging QA benchmarks and four reasoning benchmarks. The experimental results show that BNF achieves comparable performance to the best methods on QA benchmarks, while its performance decrease on the four reasoning benchmarks is significantly lower compared to the best methods, thus striking a better balance between value alignment and reasoning ability. In addition, we further validate the performance of BNF on non-pairwise datasets, and conduct in-depth analysis of log-likelihood and logit shifts across different preference optimization methods.