An Ensemble Teacher-Student Learning Approach with Poisson Sub-sampling to Differential Privacy Preserving Speech Recognition

TL;DR

This paper introduces an ensemble teacher-student learning framework with Poisson sub-sampling that enhances differential privacy in speech recognition, achieving better privacy-utility trade-offs than existing methods.

Contribution

It presents a novel combination of Poisson sub-sampling and ensemble learning to improve differential privacy guarantees in speech recognition models.

Findings

Outperforms existing DP-preserving algorithms in speech tasks

Reduces noise needed for privacy, maintaining model accuracy

Effective in both spoken command and continuous speech recognition

Abstract

We propose an ensemble learning framework with Poisson sub-sampling to effectively train a collection of teacher models to issue some differential privacy (DP) guarantee for training data. Through boosting under DP, a student model derived from the training data suffers little model degradation from the models trained with no privacy protection. Our proposed solution leverages upon two mechanisms, namely: (i) a privacy budget amplification via Poisson sub-sampling to train a target prediction model that requires less noise to achieve a same level of privacy budget, and (ii) a combination of the sub-sampling technique and an ensemble teacher-student learning framework that introduces DP-preserving noise at the output of the teacher models and transfers DP-preserving properties via noisy labels. Privacy-preserving student models are then trained with the noisy labels to learn the knowledge with DP-protection from the teacher model ensemble. Experimental evidences on spoken command recognition and continuous speech recognition of Mandarin speech show that our proposed framework greatly outperforms existing DP-preserving algorithms in both speech processing tasks.