Team-Based Self-Play With Dual Adaptive Weighting for Fine-Tuning LLMs

TL;DR

The paper introduces TPAW, a novel self-play algorithm with adaptive weighting mechanisms that enhances the alignment of large language models through fully self-supervised training, outperforming existing methods.

Contribution

It proposes a team-based self-play framework with dual adaptive weighting to improve LLM alignment without human supervision.

Findings

TPAW outperforms existing baselines across various models and benchmarks.

Adaptive weighting mechanisms improve training stability and response quality.

The method reduces reliance on human-labeled data for alignment.

Abstract

While recent self-training approaches have reduced reliance on human-labeled data for aligning LLMs, they still face critical limitations: (i) sensitivity to synthetic data quality, leading to instability and bias amplification in iterative training; (ii) ineffective optimization due to a diminishing gap between positive and negative responses over successive training iterations. In this paper, we propose Team-based self-Play with dual Adaptive Weighting (TPAW), a novel self-play algorithm designed to improve alignment in a fully self-supervised setting. TPAW adopts a team-based framework in which the current policy model both collaborates with and competes against historical checkpoints, promoting more stable and efficient optimization. To further enhance learning, we design two adaptive weighting mechanisms: (i) a response reweighting scheme that adjusts the importance of target responses, and (ii) a player weighting strategy that dynamically modulates each team member's contribution during training. Initialized from a SFT model, TPAW iteratively refines alignment without requiring additional human supervision. Experimental results demonstrate that TPAW consistently outperforms existing baselines across various base models and LLM benchmarks. Our code is publicly available at https://github.com/lab-klc/TPAW.