Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment

TL;DR

This paper introduces FIGA, an improved alignment method for large language models that uses fine-grained quality signals from contrasting responses, enhancing alignment beyond imitation learning.

Contribution

The paper presents a new dataset and a novel loss function that incorporate token-level quality signals for better alignment of LLMs.

Findings

FIGA outperforms baseline methods in alignment tasks.

Fine-grained signals improve the understanding of expected behaviors.

The approach enhances LLM alignment without relying solely on imitation learning.

Abstract

Alignment with human preference is a desired property of large language models (LLMs). Currently, the main alignment approach is based on reinforcement learning from human feedback (RLHF). Despite the effectiveness of RLHF, it is intricate to implement and train, thus recent studies explore how to develop alternative alignment approaches based on supervised fine-tuning (SFT). A major limitation of SFT is that it essentially does imitation learning, which cannot fully understand what are the expected behaviors. To address this issue, we propose an improved alignment approach named FIGA. Different from prior methods, we incorporate fine-grained (i.e., token or phrase level) quality signals that are derived by contrasting good and bad responses. Our approach has made two major contributions. Firstly, we curate a refined alignment dataset that pairs initial responses and the corresponding revised ones. Secondly, we devise a new loss function can leverage fine-grained quality signals to instruct the learning of LLMs for alignment. Extensive experiments have demonstrated the effectiveness of our approaches by comparing a number of competitive baselines.