Semi-supervised Fine-tuning for Large Language Models

TL;DR

This paper introduces SemiEvol, a semi-supervised fine-tuning framework for large language models that effectively leverages both labeled and unlabeled data to improve domain-specific performance.

Contribution

The paper proposes SemiEvol, a novel semi-supervised fine-tuning approach with a propagate-and-select mechanism for better knowledge transfer in LLMs.

Findings

SemiEvol outperforms traditional SFT and self-evolution methods.

Significant performance improvements on seven datasets.

Effective in hybrid labeled and unlabeled data scenarios.

Abstract

Supervised fine-tuning (SFT) is crucial in adapting large language model (LLMs) to a specific domain or task. However, only a limited amount of labeled data is available in practical applications, which poses a severe challenge for SFT in yielding satisfactory results. Therefore, a data-efficient framework that can fully exploit labeled and unlabeled data for LLM fine-tuning is highly anticipated.Towards this end, we introduce a semi-supervised fine-tuning(SemiFT) task and a framework named SemiEvol for LLM alignment from a propagate-and-select manner. For knowledge propagation, SemiEvol adopts a bi-level approach, propagating knowledge from labeled data to unlabeled data through both in-weight and in-context methods. For knowledge selection, SemiEvol incorporates a collaborative learning mechanism, selecting higher-quality pseudo-response samples. We conducted experiments using GPT-4o-mini and Llama-3.1 on seven general or domain-specific datasets, demonstrating significant improvements in model performance on target data. Furthermore, we compared SemiEvol with SFT and self-evolution methods, highlighting its practicality in hybrid data scenarios.