Language Models Resist Alignment: Evidence From Data Compression

TL;DR

This paper investigates the resilience of large language models to alignment efforts, revealing that fine-tuning effects are often temporary and models tend to revert to pre-training behaviors, especially as models grow larger.

Contribution

It provides the first combined theoretical and empirical analysis showing the elasticity of LLMs and how fine-tuning impacts alignment, highlighting the challenges in achieving robust alignment.

Findings

Models tend to revert to pre-training behavior after fine-tuning.

Elasticity increases with model size and pre-training data.

Fine-tuning's impact diminishes over time, favoring pre-training distribution.

Abstract

Large language models (LLMs) may exhibit unintended or undesirable behaviors. Recent works have concentrated on aligning LLMs to mitigate harmful outputs. Despite these efforts, some anomalies indicate that even a well-conducted alignment process can be easily circumvented, whether intentionally or accidentally. Does alignment fine-tuning yield have robust effects on models, or are its impacts merely superficial? In this work, we make the first exploration of this phenomenon from both theoretical and empirical perspectives. Empirically, we demonstrate the $\mathbf{elasticity}$ of post-alignment models, i.e., the tendency to revert to the behavior distribution formed during the pre-training phase upon further fine-tuning. Leveraging compression theory, we formally deduce that fine-tuning disproportionately undermines alignment relative to pre-training, potentially by orders of magnitude. We validate the presence of elasticity through experiments on models of varying types and scales. Specifically, we find that model performance declines rapidly before reverting to the pre-training distribution, after which the rate of decline drops significantly. Furthermore, we further reveal that elasticity positively correlates with the increased model size and the expansion of pre-training data. Our findings underscore the need to address the inherent elasticity of LLMs to mitigate their resistance to alignment. The model weight and code are available at pku-lm-resist-alignment.github.io.