An Uncertainty-Driven Adaptive Self-Alignment Framework for Large Language Models

TL;DR

This paper introduces UDASA, an automated framework that enhances large language model alignment by quantifying output uncertainty and progressively optimizing responses across different confidence stages.

Contribution

The paper presents a novel uncertainty-driven self-alignment framework that improves LLM alignment without human annotations by dynamically categorizing training samples based on uncertainty.

Findings

UDASA outperforms existing methods in alignment tasks

Significant improvements in harmlessness, helpfulness, and truthfulness

Effective in controlled sentiment generation

Abstract

Large Language Models (LLMs) have demonstrated remarkable progress in instruction following and general-purpose reasoning. However, achieving high-quality alignment with human intent and safety norms without human annotations remains a fundamental challenge. In this work, we propose an Uncertainty-Driven Adaptive Self-Alignment (UDASA) framework designed to improve LLM alignment in a fully automated manner. UDASA first generates multiple responses for each input and quantifies output uncertainty across three dimensions: semantics, factuality, and value alignment. Based on these uncertainty scores, the framework constructs preference pairs and categorizes training samples into three stages, conservative, moderate, and exploratory, according to their uncertainty difference. The model is then optimized progressively across these stages. In addition, we conduct a series of preliminary studies to validate the core design assumptions and provide strong empirical motivation for the proposed framework. Experimental results show that UDASA outperforms existing alignment methods across multiple tasks, including harmlessness, helpfulness, truthfulness, and controlled sentiment generation, significantly improving model performance.