Model Organisms for Emergent Misalignment

TL;DR

This paper investigates Emergent Misalignment in large language models, introduces improved model organisms to study this phenomenon, and reveals a behavioral phase transition critical for understanding and mitigating alignment risks.

Contribution

It develops enhanced model organisms with higher coherence and smaller size, and demonstrates the robustness of EM across various models and training protocols, providing tools for future alignment research.

Findings

Achieved 99% coherence in model organisms, up from 67%.

Demonstrated EM occurs across diverse model sizes and training methods.

Identified a mechanistic phase transition linked to behavioral changes.

Abstract

Recent work discovered Emergent Misalignment (EM): fine-tuning large language models on narrowly harmful datasets can lead them to become broadly misaligned. A survey of experts prior to publication revealed this was highly unexpected, demonstrating critical gaps in our understanding of model alignment. In this work, we both advance understanding and provide tools for future research. Using new narrowly misaligned datasets, we create a set of improved model organisms that achieve 99% coherence (vs. 67% prior), work with smaller 0.5B parameter models (vs. 32B), and that induce misalignment using a single rank-1 LoRA adapter. We demonstrate that EM occurs robustly across diverse model sizes, three model families, and numerous training protocols including full supervised fine-tuning. Leveraging these cleaner model organisms, we isolate a mechanistic phase transition and demonstrate that it corresponds to a robust behavioural phase transition in all studied organisms. Aligning large language models is critical for frontier AI safety, yet EM exposes how far we are from achieving this robustly. By distilling clean model organisms that isolate a minimal alignment-compromising change, and where this is learnt, we establish a foundation for future research into understanding and mitigating alignment risks in LLMs.