Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges

TL;DR

This survey reviews the current landscape of large language model alignment, discussing techniques, challenges, and evaluation methods to ensure models align with human values and intentions.

Contribution

It provides a comprehensive overview of alignment methods, paradigms, and evaluation frameworks, highlighting recent advances and open challenges in LLM alignment.

Findings

Supervised fine-tuning enables basic instruction-following.

Preference-based methods offer nuanced alignment with human intent.

Current evaluation frameworks face limitations like reward misspecification.

Abstract

Due to the remarkable capabilities and growing impact of large language models (LLMs), they have been deeply integrated into many aspects of society. Thus, ensuring their alignment with human values and intentions has emerged as a critical challenge. This survey provides a comprehensive overview of practical alignment techniques, training protocols, and empirical findings in LLM alignment. We analyze the development of alignment methods across diverse paradigms, characterizing the fundamental trade-offs between core alignment objectives. Our analysis shows that while supervised fine-tuning enables basic instruction-following, preference-based methods offer more flexibility for aligning with nuanced human intent. We discuss state-of-the-art techniques, including Direct Preference Optimization (DPO), Constitutional AI, brain-inspired methods, and alignment uncertainty quantification (AUQ), highlighting their approaches to balancing quality and efficiency. We review existing evaluation frameworks and benchmarking datasets, emphasizing limitations such as reward misspecification, distributional robustness, and scalable oversight. We summarize strategies adopted by leading AI labs to illustrate the current state of practice. We conclude by outlining open problems in oversight, value pluralism, robustness, and continuous alignment. This survey aims to inform both researchers and practitioners navigating the evolving landscape of LLM alignment.