Towards Watermarking of Open-Source LLMs

TL;DR

This paper addresses the challenge of watermarking open-source large language models, proposing a systematic evaluation framework and highlighting the durability issues of existing methods against common model modifications.

Contribution

It formulates key requirements for open-source LLM watermarking, introduces an evaluation setup, and assesses existing methods, revealing their lack of durability.

Findings

Existing watermarking methods are not durable against model modifications.

Durability is essential for effective open-source watermarking.

The paper provides a foundation for future research in this area.

Abstract

While watermarks for closed LLMs have matured and have been included in large-scale deployments, these methods are not applicable to open-source models, which allow users full control over the decoding process. This setting is understudied yet critical, given the rising performance of open-source models. In this work, we lay the foundation for systematic study of open-source LLM watermarking. For the first time, we explicitly formulate key requirements, including durability against common model modifications such as model merging, quantization, or finetuning, and propose a concrete evaluation setup. Given the prevalence of these modifications, durability is crucial for an open-source watermark to be effective. We survey and evaluate existing methods, showing that they are not durable. We also discuss potential ways to improve their durability and highlight remaining challenges. We hope our work enables future progress on this important problem.