P-Aligner: Enabling Pre-Alignment of Language Models via Principled Instruction Synthesis

TL;DR

P-Aligner is a lightweight instruction synthesis module that pre-aligns language models with human preferences, improving safety and helpfulness without costly retraining.

Contribution

The paper introduces P-Aligner, a novel, efficient method for pre-aligning language models using a new dataset and principled instruction synthesis, outperforming existing approaches.

Findings

P-Aligner achieves significant win-rate improvements on benchmarks.

It outperforms strong baselines across various models.

The method is validated for efficiency and effectiveness.

Abstract

Large Language Models (LLMs) are expected to produce safe, helpful, and honest content during interaction with human users, but they frequently fail to align with such values when given flawed instructions, e.g., missing context, ambiguous directives, or inappropriate tone, leaving substantial room for improvement along multiple dimensions. A cost-effective yet high-impact way is to pre-align instructions before the model begins decoding. Existing approaches either rely on prohibitive test-time search costs or end-to-end model rewrite, which is powered by a customized training corpus with unclear objectives. In this work, we demonstrate that the goal of efficient and effective preference alignment can be achieved by P-Aligner, a lightweight module generating instructions that preserve the original intents while being expressed in a more human-preferred form. P-Aligner is trained on UltraPrompt, a new dataset synthesized via a proposed principle-guided pipeline using Monte-Carlo Tree Search, which systematically explores the space of candidate instructions that are closely tied to human preference. Experiments across different methods show that P-Aligner generally outperforms strong baselines across various models and benchmarks, including average win-rate gains of 28.35% and 8.69% on GPT-4-turbo and Gemma-2-SimPO, respectively. Further analyses validate its effectiveness and efficiency through multiple perspectives, including data quality, search strategies, iterative deployment, and time overhead.