SSG: Logit-Balanced Vocabulary Partitioning for LLM Watermarking

TL;DR

This paper introduces SSG, a vocabulary partitioning method that enhances watermark detectability in large language models by increasing watermark strength through logit-balanced splits.

Contribution

The paper proposes SSG, a novel vocabulary partitioning algorithm that improves watermarking effectiveness by raising the lower bound of watermark strength in LLMs.

Findings

SSG improves watermark detectability in code and math reasoning tasks.

Experiments show increased watermark strength with SSG compared to random partitioning.

SSG effectively enhances watermark robustness under low-entropy conditions.

Abstract

Watermarking has emerged as a promising technique for tracing the authorship of content generated by large language models (LLMs). Among existing approaches, the KGW scheme is particularly attractive due to its versatility, efficiency, and effectiveness in natural language generation. However, KGW's effectiveness degrades significantly under low-entropy settings such as code generation and mathematical reasoning. A crucial step in the KGW method is random vocabulary partitioning, which enables adjustments to token selection based on specific preferences. Our study revealed that the next-token probability distribution plays an critical role in determining how much, or even whether, we can modify token selection and, consequently, the effectiveness of watermarking. We refer to this characteristic, associated with the probability distribution of each token prediction, as \emph{watermark strength.} In cases of random vocabulary partitioning, the lower bound of watermark strength is dictated by the next-token probability distribution. However, we found that, by redesigning the vocabulary partitioning algorithm, we can potentially raise this lower bound. In this paper, we propose SSG (\textbf{S}ort-then-\textbf{S}plit by \textbf{G}roups), a method that partitions the vocabulary into two logit-balanced subsets. This design lifts the lower bound of watermark strength for each token prediction, thereby improving watermark detectability. Experiments on code generation and mathematical reasoning datasets demonstrate the effectiveness of SSG.