LLM Safety Alignment is Divergence Estimation in Disguise

TL;DR

This paper introduces a theoretical framework that interprets LLM alignment techniques as divergence estimators, explains safety separation in latent space, and proposes a new KL divergence-based alignment method validated through empirical experiments.

Contribution

It provides a novel divergence-based perspective on LLM alignment methods and introduces KLDO, a new divergence-based alignment technique with empirical validation.

Findings

Alignment methods can be viewed as divergence estimators.

KLDO improves safety alignment performance.

Distance metrics in prompt space correlate with model safety.

Abstract

We present a theoretical framework showing that popular LLM alignment methods, including RLHF and its variants, can be understood as divergence estimators between aligned (safe or preferred) and unaligned (harmful or less preferred) distributions. This perspective explains the emergence of separation in the latent space between safe and harmful prompts after alignment. As an application of our general divergence framework, we propose KLDO, a novel KL divergence-based alignment method, and empirically validate its effectiveness. We further show that using compliance-refusal datasets, rather than standard preference-based datasets, leads to stronger separation and improved safety alignment. Finally, to quantify the separation effect, we propose a distance-based metric in the prompt representation space, which also acts as a statistically significant indicator for model safety.