ASFT: Aligned Supervised Fine-Tuning through Absolute Likelihood

TL;DR

This paper introduces ASFT, a novel fine-tuning method that improves alignment of large language models with human preferences by optimizing absolute likelihood, addressing limitations of DPO.

Contribution

ASFT is a new fine-tuning approach that optimizes absolute likelihood for better alignment, eliminating the need for a reference model and mitigating issues in DPO.

Findings

ASFT outperforms DPO and variants on instruction-following benchmarks.

Theoretical analysis shows ASFT mitigates probability decrease of dispreferred data.

Extensive experiments confirm ASFT's effectiveness in model alignment.

Abstract

Direct Preference Optimization (DPO) is a method for enhancing model performance by directly optimizing for the preferences or rankings of outcomes, instead of traditional loss functions. This approach has proven effective in aligning Large Language Models (LLMs) with human preferences. Despite its widespread use across various tasks, DPO has been criticized for its sensitivity to the effectiveness of Supervised Fine-Tuning (SFT) and its limitations in enabling models to learn human-preferred responses, leading to less satisfactory performance. To address these limitations, we propose Aligned Supervised Fine-Tuning (ASFT), an effective approach that better aligns LLMs with pair-wise datasets by optimizing absolute likelihood for each response, rather than using the Bradley-Terry model, and eliminates the need for a reference model. Through theoretical gradient analysis, we demonstrate that ASFT mitigates the issue where the DPO loss function decreases the probability of generating human-dispreferred data at a faster rate than it increases the probability of producing preferred data. Additionally, we compare ASFT to DPO and its latest variants, such as the single-step approach ORPO, using the latest instruction-tuned model Llama3, which has been fine-tuned on UltraFeedback and HH-RLHF. We evaluated performance on instruction-following benchmarks like MT-Bench and traditional text generation metrics such as BLEU-4 and ROUGE-L. Extensive experiments demonstrate that ASFT is an effective alignment approach, consistently outperforming existing methods.