TokenMark: A Modality-Agnostic Watermark for Pre-trained Transformers

TL;DR

TokenMark introduces a universal, robust watermarking method for pre-trained Transformer models that leverages permutation properties, enabling effective ownership verification across different data modalities.

Contribution

It proposes TokenMark, a modality-agnostic watermarking system that embeds watermarks by fine-tuning with permuted data, enhancing robustness and universality.

Findings

Significantly improves watermark robustness against attacks.

Works effectively across various pre-trained Transformer models.

Maintains model performance while embedding watermarks.

Abstract

Watermarking is a critical tool for model ownership verification. However, existing watermarking techniques are often designed for specific data modalities and downstream tasks, without considering the inherent architectural properties of the model. This lack of generality and robustness underscores the need for a more versatile watermarking approach. In this work, we investigate the properties of Transformer models and propose TokenMark, a modality-agnostic, robust watermarking system for pre-trained models, leveraging the permutation equivariance property. TokenMark embeds the watermark by fine-tuning the pre-trained model on a set of specifically permuted data samples, resulting in a watermarked model that contains two distinct sets of weights -- one for normal functionality and the other for watermark extraction, the latter triggered only by permuted inputs. Extensive experiments on state-of-the-art pre-trained models demonstrate that TokenMark significantly improves the robustness, efficiency, and universality of model watermarking, highlighting its potential as a unified watermarking solution.