Canary Extraction in Natural Language Understanding Models

TL;DR

This paper demonstrates how adversaries can extract sensitive canary information from NLU models using model inversion attacks, and proposes defense strategies to mitigate this privacy risk.

Contribution

It introduces a novel canary extraction attack in NLU models and evaluates effective countermeasures to prevent data leakage.

Findings

Attack successfully reconstructs canaries with 50% probability.

Combining multiple defenses effectively prevents canary extraction.

The attack works with open-box access and prefix-based text completion.

Abstract

Natural Language Understanding (NLU) models can be trained on sensitive information such as phone numbers, zip-codes etc. Recent literature has focused on Model Inversion Attacks (ModIvA) that can extract training data from model parameters. In this work, we present a version of such an attack by extracting canaries inserted in NLU training data. In the attack, an adversary with open-box access to the model reconstructs the canaries contained in the model's training set. We evaluate our approach by performing text completion on canaries and demonstrate that by using the prefix (non-sensitive) tokens of the canary, we can generate the full canary. As an example, our attack is able to reconstruct a four digit code in the training dataset of the NLU model with a probability of 0.5 in its best configuration. As countermeasures, we identify several defense mechanisms that, when combined, effectively eliminate the risk of ModIvA in our experiments.