Advantage-Guided Distillation for Preference Alignment in Small Language Models

TL;DR

This paper introduces a novel approach to improve the alignment of Small Language Models by leveraging a well-aligned teacher model through advantage-guided distillation, significantly narrowing the performance gap with larger models.

Contribution

It proposes Advantage-Guided Distillation for Preference Alignment (ADPA), a new method that transfers human preference knowledge from a large teacher LLM to small models, enhancing their alignment.

Findings

ADPA outperforms existing methods in aligning SLMs with human preferences.

Combining ADPA with DCKD yields even better alignment results.

The approaches significantly reduce the performance gap between small and large language models.

Abstract

Alignment techniques enable Large Language Models (LLMs) to generate outputs that align with human preferences and play a crucial role in their effectiveness. However, their impact often diminishes when applied to Small Language Models (SLMs), likely due to the limited capacity of these models. Instead of directly applying existing alignment techniques to SLMs, we propose to utilize a well-aligned teacher LLM to guide the alignment process for these models, thereby facilitating the transfer of the teacher's knowledge of human preferences to the student model. To achieve this, we first explore a straightforward approach, Dual-Constrained Knowledge Distillation (DCKD), that employs knowledge distillation with two KL-divergence constraints from the aligned teacher to the unaligned student. To further enhance the student's ability to distinguish between preferred and dispreferred responses, we then propose Advantage-Guided Distillation for Preference Alignment (ADPA), which leverages an advantage function from the aligned teacher to deliver more nuanced, distribution-level reward signals for the student's alignment. Our experimental results show that these two approaches appreciably improve the alignment of SLMs and narrow the performance gap with larger counterparts. Among them, ADPA demonstrates superior performance and achieves even greater effectiveness when integrated with DCKD. Our code is available at https://github.com/SLIT-AI/ADPA.