BiMark: Unbiased Multilayer Watermarking for Large Language Models

TL;DR

BiMark introduces an innovative multilayer watermarking framework for large language models that balances text quality, detection robustness, and message capacity, outperforming existing methods in extraction rate and maintaining downstream task performance.

Contribution

The paper presents BiMark, a novel multilayer watermarking approach with unbiased reweighting and multi-bit encoding, advancing the state-of-the-art in LLM text authentication.

Findings

Up to 30% higher extraction rates for short texts.

Lower perplexity indicates better text quality.

Maintains performance on downstream tasks like summarization and translation.

Abstract

Recent advances in Large Language Models (LLMs) have raised urgent concerns about LLM-generated text authenticity, prompting regulatory demands for reliable identification mechanisms. Although watermarking offers a promising solution, existing approaches struggle to simultaneously achieve three critical requirements: text quality preservation, model-agnostic detection, and message embedding capacity, which are crucial for practical implementation. To achieve these goals, the key challenge lies in balancing the trade-off between text quality preservation and message embedding capacity. To address this challenge, we propose BiMark, a novel watermarking framework that achieves these requirements through three key innovations: (1) a bit-flip unbiased reweighting mechanism enabling model-agnostic detection, (2) a multilayer architecture enhancing detectability without compromising generation quality, and (3) an information encoding approach supporting multi-bit watermarking. Through theoretical analysis and extensive experiments, we validate that, compared to state-of-the-art multi-bit watermarking methods, BiMark achieves up to 30% higher extraction rates for short texts while maintaining text quality indicated by lower perplexity, and performs comparably to non-watermarked text on downstream tasks such as summarization and translation.