A Reinforcement Learning Framework for Robust and Secure LLM Watermarking

TL;DR

This paper introduces an end-to-end reinforcement learning framework for watermarking large language models, balancing detectability, quality, robustness, and security, and outperforming existing methods in resisting spoofing attacks.

Contribution

The paper presents a novel RL-based watermarking method with an anchoring mechanism and regularization to enhance stability and security, addressing challenges of reward hacking and multi-criteria optimization.

Findings

Achieves state-of-the-art trade-offs across multiple watermarking criteria.

Improves resistance to spoofing attacks without sacrificing text quality.

Demonstrates effectiveness on standard benchmarks with two LLMs.

Abstract

Watermarking has emerged as a promising solution for tracing and authenticating text generated by large language models (LLMs). A common approach to LLM watermarking is to construct a green/red token list and assign higher or lower generation probabilities to the corresponding tokens, respectively. However, most existing watermarking algorithms rely on heuristic green/red token list designs, as directly optimizing the list design with techniques such as reinforcement learning (RL) comes with several challenges. First, desirable watermarking involves multiple criteria, i.e., detectability, text quality, robustness against removal attacks, and security against spoofing attacks. Directly optimizing for these criteria introduces many partially conflicting reward terms, leading to an unstable convergence process. Second, the vast action space of green/red token list choices is susceptible to reward hacking. In this paper, we propose an end-to-end RL framework for robust and secure LLM watermarking. Our approach adopts an anchoring mechanism for reward terms to ensure stable training and introduces additional regularization terms to prevent reward hacking. Experiments on standard benchmarks with two backbone LLMs show that our method achieves a state-of-the-art trade-off across all criteria, with notable improvements in resistance to spoofing attacks without degrading other criteria. Our code is available at https://github.com/UCSB-NLP-Chang/RL-watermark.