Analyzing Information Leakage of Updates to Natural Language Models

TL;DR

This paper demonstrates that analyzing differences between language model snapshots before and after updates can reveal detailed information about training data changes, raising privacy concerns and suggesting mitigation strategies.

Contribution

It introduces two new metrics, differential score and differential rank, for analyzing information leakage during model updates in natural language models.

Findings

Differential analysis reveals significant data change information.

Proposed metrics effectively quantify information leakage.

Mitigation strategies can reduce leakage impact.

Abstract

To continuously improve quality and reflect changes in data, machine learning applications have to regularly retrain and update their core models. We show that a differential analysis of language model snapshots before and after an update can reveal a surprising amount of detailed information about changes in the training data. We propose two new metrics---\emph{differential score} and \emph{differential rank}---for analyzing the leakage due to updates of natural language models. We perform leakage analysis using these metrics across models trained on several different datasets using different methods and configurations. We discuss the privacy implications of our findings, propose mitigation strategies and evaluate their effect.