Watermarking Degrades Alignment in Language Models: Analysis and Mitigation

TL;DR

Watermarking language models causes alignment issues, but a simple sampling method called Alignment Resampling can mitigate these effects and restore model safety and helpfulness.

Contribution

This paper provides the first empirical analysis of how watermarking impacts model alignment and introduces Alignment Resampling as an effective mitigation technique.

Findings

Watermarking induces model-specific shifts in alignment.

Sampling multiple outputs improves alignment performance.

Alignment Resampling restores safety and helpfulness without harming watermark detection.

Abstract

Watermarking has become a practical tool for tracing language model outputs, but it modifies token probabilities at inference time, which were carefully tuned by alignment training. This creates a tension: how do watermark-induced shifts interact with the procedures intended to make models safe and useful? Experiments on several contemporary models and two representative watermarking schemes reveal that watermarking induces a nontrivial, patterned yet model-specific shift in alignment. We see two failure modes: guard attenuation, where models become more helpful but less safe, and guard amplification, where refusals become overly conservative. These effects persist even after controlling for perplexity degradation, pointing to alignment-specific distortions, not just quality loss. We address this with Alignment Resampling (AR), a procedure that samples multiple watermarked outputs and selects the most aligned response according to an external reward model. Using standard results on the expected maximum of Gaussian random variables, we derive a theoretical lower bound showing that alignment gains grow sublogarithmically with sample size. In practice, sampling as few as two to four candidates largely restores unwatermarked alignment performance in truthfulness, safety, and helpfulness, without hurting watermark detection. This is the first empirical study of watermarking-alignment interactions; it shows that a simple inference-time fix can recover alignment.