Direct Alignment of Language Models via Quality-Aware Self-Refinement

TL;DR

This paper introduces a quality-aware self-refinement method that enhances direct policy optimization for aligning large language models with human preferences by leveraging the model's intrinsic knowledge.

Contribution

It proposes a novel refinement function that uses the LLM's own knowledge to improve the training process of DPO and IPO, leading to better alignment results.

Findings

Improved model performance over standard DPO and IPO.

Effective use of intrinsic LLM knowledge for self-refinement.

Enhanced alignment with human preferences.

Abstract

Reinforcement Learning from Human Feedback (RLHF) has been commonly used to align the behaviors of Large Language Models (LLMs) with human preferences. Recently, a popular alternative is Direct Policy Optimization (DPO), which replaces an LLM-based reward model with the policy itself, thus obviating the need for extra memory and training time to learn the reward model. However, DPO does not consider the relative qualities of the positive and negative responses, and can lead to sub-optimal training outcomes. To alleviate this problem, we investigate the use of intrinsic knowledge within the on-the-fly fine-tuning LLM to obtain relative qualities and help to refine the loss function. Specifically, we leverage the knowledge of the LLM to design a refinement function to estimate the quality of both the positive and negative responses. We show that the constructed refinement function can help self-refine the loss function under mild assumptions. The refinement function is integrated into DPO and its variant Identity Policy Optimization (IPO). Experiments across various evaluators indicate that they can improve the performance of the fine-tuned models over DPO and IPO.