ArcMark: Multi-bit LLM Watermark via Optimal Transport

TL;DR

This paper introduces ArcMark, a novel multi-bit watermarking method for language models based on optimal transport and coding theory, achieving maximum capacity and outperforming existing methods in detection and bit rate.

Contribution

It provides the first capacity characterization of multi-bit watermarking and designs ArcMark to reach this capacity using coding-theoretic principles.

Findings

ArcMark achieves higher bit rate per token than existing methods.

ArcMark demonstrates superior detection accuracy in experiments.

The work establishes watermarking as a channel coding problem.

Abstract

Watermarking is an important tool for promoting the responsible use of language models (LMs). Existing watermarks insert a signal into generated tokens that either flags LM-generated text (zero-bit watermarking) or encodes more complex messages (multi-bit watermarking). Though a number of recent multi-bit watermarks insert several bits into text without perturbing average next-token predictions, they largely extend design principles from the zero-bit setting, such as encoding a single bit per token. Notably, the information-theoretic capacity of multi-bit watermarking -- the maximum number of bits per token that can be inserted and detected without changing average next-token predictions -- has remained unknown. We address this gap by deriving the first capacity characterization of multi-bit watermarks. Our results inform the design of ArcMark: a new watermark construction based on coding-theoretic principles that, under certain assumptions, achieves the capacity of the multi-bit watermark channel. In practice, ArcMark outperforms competing multi-bit watermarks in terms of bit rate per token and detection accuracy. Our work demonstrates that LM watermarking is fundamentally a channel coding problem, paving the way for principled coding-theoretic approaches to watermark design.