Self-Generative Adversarial Fine-Tuning for Large Language Models

TL;DR

This paper introduces SGALM, a novel framework that uses a generative adversarial approach within a single large language model to improve alignment without external rewards, achieving state-of-the-art results.

Contribution

SGALM is the first unified adversarial fine-tuning method that jointly evolves generation and discrimination in a single LLM for alignment tasks.

Findings

Achieves state-of-the-art alignment performance.

Operates without external reward models.

Serves as a robust synthetic data generator.

Abstract

Fine-tuning large language models (LLMs) for alignment typically relies on supervised fine-tuning or reinforcement learning from human feedback, both limited by the cost and scarcity of high-quality annotations. Recent self-play and synthetic data approaches reduce this dependence but often rely on heuristic assumptions or ungrounded self-evaluation, which can cause bias accumulation and performance drift. In this paper, we propose Self-Generative Adversarial LLM (SGALM), a unified fine-tuning framework that formulates alignment as a generative adversarial game within a single LLM. SGALM jointly evolves generation and discrimination capabilities without external reward models. Theoretical and empirical results demonstrate that SGALM achieves state-of-the-art performance, serves as an effective alignment algorithm and a robust synthetic data engine.