Distillation-Resistant Watermarking for Model Protection in NLP

TL;DR

This paper introduces Distillation-Resistant Watermarking (DRW), a novel method to protect NLP models from theft via distillation by embedding detectable watermarks into model predictions without significantly affecting accuracy.

Contribution

The paper presents DRW, the first watermarking technique specifically designed for NLP models that resists model distillation and effectively detects stolen models.

Findings

DRW achieves 100% detection precision across multiple NLP tasks.

DRW maintains original model accuracy within a certain bound.

Prior methods fail to detect theft in some tasks.

Abstract

How can we protect the intellectual property of trained NLP models? Modern NLP models are prone to stealing by querying and distilling from their publicly exposed APIs. However, existing protection methods such as watermarking only work for images but are not applicable to text. We propose Distillation-Resistant Watermarking (DRW), a novel technique to protect NLP models from being stolen via distillation. DRW protects a model by injecting watermarks into the victim's prediction probability corresponding to a secret key and is able to detect such a key by probing a suspect model. We prove that a protected model still retains the original accuracy within a certain bound. We evaluate DRW on a diverse set of NLP tasks including text classification, part-of-speech tagging, and named entity recognition. Experiments show that DRW protects the original model and detects stealing suspects at 100% mean average precision for all four tasks while the prior method fails on two.