Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs

TL;DR

The paper introduces T-SPIN, a novel self-play fine-tuning method for large language models that stabilizes training and improves performance, especially with limited annotated data, by using triplet-based advantages and entropy constraints.

Contribution

T-SPIN enhances self-play fine-tuning by incorporating historical advantages and entropy constraints, addressing stability and alignment issues in previous methods.

Findings

T-SPIN outperforms SPIN in various tasks.

T-SPIN achieves comparable or better results than supervised fine-tuning with only 25% of labeled data.

T-SPIN demonstrates stable iterative evolution during training.

Abstract

Recently, self-play fine-tuning (SPIN) has been proposed to adapt large language models to downstream applications with scarce expert-annotated data, by iteratively generating synthetic responses from the model itself. However, SPIN is designed to optimize the current reward advantages of annotated responses over synthetic responses at hand, which may gradually vanish during iterations, leading to unstable optimization. Moreover, the utilization of reference policy induces a misalignment issue between the reward formulation for training and the metric for generation. To address these limitations, we propose a novel Triplet-based Self-Play fIne-tuNing (T-SPIN) method that integrates two key designs. First, beyond current advantages, T-SPIN additionally incorporates historical advantages between iteratively generated responses and proto-synthetic responses produced by the initial policy. Even if the current advantages diminish, historical advantages remain effective, stabilizing the overall optimization. Second, T-SPIN introduces the entropy constraint into the self-play framework, which is theoretically justified to support reference-free fine-tuning, eliminating the training-generation discrepancy. Empirical results on various tasks demonstrate not only the superior performance of T-SPIN over SPIN, but also its stable evolution during iterations. Remarkably, compared to supervised fine-tuning, T-SPIN achieves comparable or even better performance with only 25% samples, highlighting its effectiveness when faced with scarce annotated data.