Learning to Align, Aligning to Learn: A Unified Approach for Self-Optimized Alignment

TL;DR

This paper introduces GRAO, a unified framework combining supervised fine-tuning and reinforcement learning to improve language model alignment, achieving significant performance gains and theoretical guarantees.

Contribution

GRAO is a novel unified approach that integrates SFT and RL for better alignment, with new strategies for sample generation, loss formulation, and parameter updates.

Findings

GRAO outperforms SFT, DPO, PPO, and GRPO baselines in alignment tasks.

Theoretical analysis confirms GRAO's convergence and sample efficiency.

Empirical results show substantial relative improvements in alignment performance.

Abstract

Alignment methodologies have emerged as a critical pathway for enhancing language model alignment capabilities. While SFT (supervised fine-tuning) accelerates convergence through direct token-level loss intervention, its efficacy is constrained by offline policy trajectory. In contrast, RL(reinforcement learning) facilitates exploratory policy optimization, but suffers from low sample efficiency and stringent dependency on high-quality base models. To address these dual challenges, we propose GRAO (Group Relative Alignment Optimization), a unified framework that synergizes the respective strengths of SFT and RL through three key innovations: 1) A multi-sample generation strategy enabling comparative quality assessment via reward feedback; 2) A novel Group Direct Alignment Loss formulation leveraging intra-group relative advantage weighting; 3) Reference-aware parameter updates guided by pairwise preference dynamics. Our theoretical analysis establishes GRAO's convergence guarantees and sample efficiency advantages over conventional approaches. Comprehensive evaluations across complex human alignment tasks demonstrate GRAO's superior performance, achieving 57.70\%,17.65\% 7.95\% and 5.18\% relative improvements over SFT, DPO, PPO and GRPO baselines respectively. This work provides both a theoretically grounded alignment framework and empirical evidence for efficient capability evolution in language models.