Optimal Estimation of Watermark Proportions in Hybrid AI-Human Texts

TL;DR

This paper develops optimal methods for estimating the proportion of watermarked content in texts that are a mixture of human and AI-generated text, addressing a key challenge in watermark detection.

Contribution

It introduces identifiable estimation techniques for watermark proportions in mixed texts using pivotal statistics, with theoretical bounds and practical algorithms.

Findings

Estimators achieve minimax optimality.

High accuracy on synthetic and real data.

Applicable to popular watermarking schemes.

Abstract

Text watermarks in large language models (LLMs) are an increasingly important tool for detecting synthetic text and distinguishing human-written content from LLM-generated text. While most existing studies focus on determining whether entire texts are watermarked, many real-world scenarios involve mixed-source texts, which blend human-written and watermarked content. In this paper, we address the problem of optimally estimating the watermark proportion in mixed-source texts. We cast this problem as estimating the proportion parameter in a mixture model based on \emph{pivotal statistics}. First, we show that this parameter is not even identifiable in certain watermarking schemes, let alone consistently estimable. In stark contrast, for watermarking methods that employ continuous pivotal statistics for detection, we demonstrate that the proportion parameter is identifiable under mild conditions. We propose efficient estimators for this class of methods, which include several popular unbiased watermarks as examples, and derive minimax lower bounds for any measurable estimator based on pivotal statistics, showing that our estimators achieve these lower bounds. Through evaluations on both synthetic data and mixed-source text generated by open-source models, we demonstrate that our proposed estimators consistently achieve high estimation accuracy.