Bypassing LLM Watermarks with Color-Aware Substitutions

TL;DR

This paper introduces SCTS, a novel color-aware attack that effectively evades LLM watermarks by strategically identifying and substituting watermarked tokens, outperforming previous methods in robustness and efficiency.

Contribution

We propose SCTS, the first color-aware attack that can reliably remove watermarks from long texts, demonstrating both theoretical and empirical effectiveness against state-of-the-art watermarking.

Findings

SCTS successfully evades watermark detection with fewer edits.

SCTS can remove watermarks from arbitrarily long texts.

Theoretical analysis confirms SCTS's effectiveness.

Abstract

Watermarking approaches are proposed to identify if text being circulated is human or large language model (LLM) generated. The state-of-the-art watermarking strategy of Kirchenbauer et al. (2023a) biases the LLM to generate specific (``green'') tokens. However, determining the robustness of this watermarking method is an open problem. Existing attack methods fail to evade detection for longer text segments. We overcome this limitation, and propose {\em Self Color Testing-based Substitution (SCTS)}, the first ``color-aware'' attack. SCTS obtains color information by strategically prompting the watermarked LLM and comparing output tokens frequencies. It uses this information to determine token colors, and substitutes green tokens with non-green ones. In our experiments, SCTS successfully evades watermark detection using fewer number of edits than related work. Additionally, we show both theoretically and empirically that SCTS can remove the watermark for arbitrarily long watermarked text.