Private Deep Learning with Teacher Ensembles

TL;DR

This paper introduces a novel privacy-preserving deep learning method using teacher ensembles, knowledge distillation, and weighted ensemble schemes, achieving better performance and privacy guarantees on image datasets.

Contribution

It proposes a new teacher ensemble approach with differential privacy, leveraging knowledge distillation and hint learning for improved privacy-preserving deep learning.

Findings

Outperforms previous methods on CIFAR10, MNIST, and SVHN datasets.

Provides provable privacy guarantees with enhanced model accuracy.

Demonstrates robustness of the weighted ensemble scheme across different settings.

Abstract

Privacy-preserving deep learning is crucial for deploying deep neural network based solutions, especially when the model works on data that contains sensitive information. Most privacy-preserving methods lead to undesirable performance degradation. Ensemble learning is an effective way to improve model performance. In this work, we propose a new method for teacher ensembles that uses more informative network outputs under differential private stochastic gradient descent and provide provable privacy guarantees. Out method employs knowledge distillation and hint learning on intermediate representations to facilitate the training of student model. Additionally, we propose a simple weighted ensemble scheme that works more robustly across different teaching settings. Experimental results on three common image datasets benchmark (i.e., CIFAR10, MINST, and SVHN) demonstrate that our approach outperforms previous state-of-the-art methods on both performance and privacy-budget.