VertMark: A Unified Training-Free Robust Watermarking Framework for Vertical Domain Pre-trained Language Models

TL;DR

VertMark introduces a training-free, robust watermarking framework for vertical domain pre-trained language models, enabling efficient copyright verification across multiple specialized fields with minimal performance impact.

Contribution

It is the first unified, training-free watermarking method that ensures scalable, robust copyright verification for VPLMs across various vertical domains.

Findings

Achieves efficient watermark embedding and verification in medical, financial, and legal VPLMs.

Maintains negligible impact on downstream task performance.

Demonstrates robustness against pruning and quantization attacks.

Abstract

With the application of vertical domain pre-trained language models (VPLMs) in specialized fields such as medical, finance, and law, model parameters and inference capabilities have become important digital assets. Achieving traceable copyright verification for VPLMs has become an urgent challenge. Existing copyright verification methods primarily rely on embedding backdoor watermarks into models. However, most of these methods require additional training, suffer from inefficient watermark embedding, and lack scalable designs for multiple vertical domains. To address these limitations, we propose VertMark, the first unified training-free and robust watermarking framework for copyright verification across multiple vertical domain VPLMs. The framework embeds ownership-encoded watermarks by establishing a hidden semantic equivalence between low-frequency trigger tokens and high-frequency domain-relevant words via a training-free parameter replacement strategy. Experiments demonstrate that VertMark can achieve efficient watermark embedding and reliable watermark verification for both text understanding and text generation downstream tasks in the medical, financial, and legal domains, with negligible impact on model performance. Moreover, VertMark exhibits strong robustness against various attacks (e.g., pruning and quantization), highlighting its practical value and providing strong protection for the copyright security of VPLMs.