Thieves on Sesame Street! Model Extraction of BERT-based APIs

TL;DR

This paper demonstrates that BERT-based NLP models can be effectively extracted using only query access without real data, highlighting security vulnerabilities in transfer learning models and evaluating defense strategies.

Contribution

It shows that model extraction is feasible with random queries and no real data, and evaluates defenses like watermarking and membership classification.

Findings

Random queries can successfully extract models.

Extraction performance is close to the original model.

Existing defenses are ineffective against sophisticated attacks.

Abstract

We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model. Assuming that both the adversary and victim model fine-tune a large pretrained language model such as BERT (Devlin et al. 2019), we show that the adversary does not need any real training data to successfully mount the attack. In fact, the attacker need not even use grammatical or semantically meaningful queries: we show that random sequences of words coupled with task-specific heuristics form effective queries for model extraction on a diverse set of NLP tasks, including natural language inference and question answering. Our work thus highlights an exploit only made feasible by the shift towards transfer learning methods within the NLP community: for a query budget of a few hundred dollars, an attacker can extract a model that performs only slightly worse than the victim model. Finally, we study two defense strategies against model extraction---membership classification and API watermarking---which while successful against naive adversaries, are ineffective against more sophisticated ones.