Self-Steering Optimization: Autonomous Preference Optimization for Large Language Models

TL;DR

This paper introduces Self-Steering Optimization ($SSO$), an autonomous method for generating high-quality preference data to improve large language model alignment without manual labeling.

Contribution

The paper presents $SSO$, a novel algorithm that autonomously produces on-policy preference data, enhancing alignment and reward optimization for large language models.

Findings

$SSO$ outperforms baselines in human preference alignment.

$SSO$ improves reward optimization across models.

The framework is scalable and effective for automated alignment.

Abstract

The key to effective alignment lies in high-quality preference data. Recent research has focused on automated alignment, which involves developing alignment systems with minimal human intervention. However, prior research has predominantly focused on developing data generation methods, while insufficient attention has been paid to quality control mechanisms, which often produce inaccurate and unhelpful data, leading to unpredictable benefits during iterative optimization. In this paper, we present Self-Steering Optimization ($SSO$), an algorithm that autonomously generates high-quality preference data, eliminating manual annotation requirements. $SSO$ employs a specialized optimization objective to build a data generator from the policy model itself, which is used to produce accurate and on-policy data. We demonstrate $SSO$'s effectiveness through comprehensive experiments on two series of models: Llama 3 and Qwen 2. Our evaluation across diverse benchmarks shows that $SSO$ consistently outperforms baselines in human preference alignment and reward optimization. Further analysis validates $SSO$ as a scalable framework for preference optimization, benefiting the advancement in automated alignment techniques.