Privacy-Preserving Model Transcription with Differentially Private Synthetic Distillation

TL;DR

This paper introduces a novel privacy-preserving model transcription method using differentially private synthetic distillation, enabling model conversion without private data access while guaranteeing privacy and maintaining performance.

Contribution

It proposes a data-free, differential privacy-guaranteed model transcriber using a cooperative-competitive learning framework with theoretical privacy and convergence proofs.

Findings

Outperforms 26 state-of-the-art methods in experiments.

Guarantees differential privacy during model transduction.

Generates private synthetic data suitable for downstream tasks.

Abstract

While many deep learning models trained on private datasets have been deployed in various practical tasks, they may pose a privacy leakage risk as attackers could recover informative data or label knowledge from models. In this work, we present \emph{privacy-preserving model transcription}, a data-free model-to-model conversion solution to facilitate model deployment with a privacy guarantee. To this end, we propose a cooperative-competitive learning approach termed \emph{differentially private synthetic distillation} that learns to convert a pretrained model (teacher) into its privacy-preserving counterpart (student) via a trainable generator without access to private data. The learning collaborates with three players in a unified framework and performs alternate optimization: i)~the generator is learned to generate synthetic data, ii)~the teacher and student accept the synthetic data and compute differential private labels by flexible data or label noisy perturbation, and iii)~the student is updated with noisy labels and the generator is updated by taking the student as a discriminator for adversarial training. We theoretically prove that our approach can guarantee differential privacy and convergence. The transcribed student has good performance and privacy protection, while the resulting generator can generate private synthetic data for downstream tasks. Extensive experiments clearly demonstrate that our approach outperforms 26 state-of-the-arts.