Privacy-Preserving Student Learning with Differentially Private Data-Free Distillation

TL;DR

This paper introduces a novel teacher-student framework for training deep learning models that preserves data privacy by generating synthetic data and applying differential privacy techniques, avoiding exposure of sensitive information.

Contribution

It proposes a data-free distillation method with a generator and a label differential privacy algorithm, enhancing privacy protection in deep learning model training.

Findings

Effective privacy preservation demonstrated through experiments

Synthetic data can replace real data for training without privacy risks

The approach maintains high model accuracy while protecting data and label privacy

Abstract

Deep learning models can achieve high inference accuracy by extracting rich knowledge from massive well-annotated data, but may pose the risk of data privacy leakage in practical deployment. In this paper, we present an effective teacher-student learning approach to train privacy-preserving deep learning models via differentially private data-free distillation. The main idea is generating synthetic data to learn a student that can mimic the ability of a teacher well-trained on private data. In the approach, a generator is first pretrained in a data-free manner by incorporating the teacher as a fixed discriminator. With the generator, massive synthetic data can be generated for model training without exposing data privacy. Then, the synthetic data is fed into the teacher to generate private labels. Towards this end, we propose a label differential privacy algorithm termed selective randomized response to protect the label information. Finally, a student is trained on the synthetic data with the supervision of private labels. In this way, both data privacy and label privacy are well protected in a unified framework, leading to privacy-preserving models. Extensive experiments and analysis clearly demonstrate the effectiveness of our approach.