Measuring Unintended Memorisation of Unique Private Features in Neural Networks

TL;DR

This paper investigates how neural networks unintentionally memorize unique private features in training data, revealing privacy risks even with common regularization strategies, especially in sensitive domains like healthcare.

Contribution

It introduces a score to estimate model sensitivity to unique features and demonstrates that standard training strategies do not prevent memorization of private information.

Findings

Neural networks memorize unique features even when they occur once.

Regularization strategies do not prevent memorization of private features.

Memorization poses privacy risks, notably in healthcare applications.

Abstract

Neural networks pose a privacy risk to training data due to their propensity to memorise and leak information. Focusing on image classification, we show that neural networks also unintentionally memorise unique features even when they occur only once in training data. An example of a unique feature is a person's name that is accidentally present on a training image. Assuming access to the inputs and outputs of a trained model, the domain of the training data, and knowledge of unique features, we develop a score estimating the model's sensitivity to a unique feature by comparing the KL divergences of the model's output distributions given modified out-of-distribution images. Our results suggest that unique features are memorised by multi-layer perceptrons and convolutional neural networks trained on benchmark datasets, such as MNIST, Fashion-MNIST and CIFAR-10. We find that strategies to prevent overfitting (e.g.\ early stopping, regularisation, batch normalisation) do not prevent memorisation of unique features. These results imply that neural networks pose a privacy risk to rarely occurring private information. These risks can be more pronounced in healthcare applications if patient information is present in the training data.