Unintended memorisation of unique features in neural networks

TL;DR

Neural networks can memorize unique, rare features in training data, posing privacy risks especially in sensitive fields like healthcare, even when standard overfitting prevention methods are used.

Contribution

This study demonstrates that neural networks memorize unique features in training data and introduces a method to estimate sensitivity to such features without access to training data.

Findings

Unique features are memorized by neural networks.

Standard overfitting prevention does not mitigate memorization.

Images with unique features are highly influential regardless of other features.

Abstract

Neural networks pose a privacy risk due to their propensity to memorise and leak training data. We show that unique features occurring only once in training data are memorised by discriminative multi-layer perceptrons and convolutional neural networks trained on benchmark imaging datasets. We design our method for settings where sensitive training data is not available, for example medical imaging. Our setting knows the unique feature, but not the training data, model weights or the unique feature's label. We develop a score estimating a model's sensitivity to a unique feature by comparing the KL divergences of the model's output distributions given modified out-of-distribution images. We find that typical strategies to prevent overfitting do not prevent unique feature memorisation. And that images containing a unique feature are highly influential, regardless of the influence the images's other features. We also find a significant variation in memorisation with training seed. These results imply that neural networks pose a privacy risk to rarely occurring private information. This risk is more pronounced in healthcare applications since sensitive patient information can be memorised when it remains in training data due to an imperfect data sanitisation process.