NoPeek: Information leakage reduction to share activations in distributed deep learning

TL;DR

This paper introduces NoPeek, a method that reduces information leakage in distributed deep learning by minimizing distance correlation between raw data and intermediate representations, thereby protecting sensitive data without sacrificing model accuracy.

Contribution

NoPeek is the first approach to effectively reduce data leakage in distributed deep learning by controlling correlation during training and inference.

Findings

Reduces risk of raw data reconstruction attacks.

Maintains high classification accuracy.

Resilient against inversion attacks.

Abstract

For distributed machine learning with sensitive data, we demonstrate how minimizing distance correlation between raw data and intermediary representations reduces leakage of sensitive raw data patterns across client communications while maintaining model accuracy. Leakage (measured using distance correlation between input and intermediate representations) is the risk associated with the invertibility of raw data from intermediary representations. This can prevent client entities that hold sensitive data from using distributed deep learning services. We demonstrate that our method is resilient to such reconstruction attacks and is based on reduction of distance correlation between raw data and learned representations during training and inference with image datasets. We prevent such reconstruction of raw data while maintaining information required to sustain good classification accuracies.