An Experimental Study on Private Aggregation of Teacher Ensemble Learning for End-to-End Speech Recognition

TL;DR

This paper extends the Private Aggregation of Teacher Ensembles (PATE) method to end-to-end speech recognition, demonstrating improved privacy-preserving accuracy on speech datasets by reducing word error rates under strict differential privacy constraints.

Contribution

It introduces a novel application of PATE to dynamic speech data, showing its effectiveness in preventing data leakage and enhancing ASR accuracy under differential privacy.

Findings

PATE-based models outperform DP-SGD in speech recognition tasks.

Significant word error rate reductions (26.2%-27.5%) under strict privacy budgets.

Proposes a DP-preserving pretraining approach for public speech data.

Abstract

Differential privacy (DP) is one data protection avenue to safeguard user information used for training deep models by imposing noisy distortion on privacy data. Such a noise perturbation often results in a severe performance degradation in automatic speech recognition (ASR) in order to meet a privacy budget $\varepsilon$. Private aggregation of teacher ensemble (PATE) utilizes ensemble probabilities to improve ASR accuracy when dealing with the noise effects controlled by small values of $\varepsilon$. We extend PATE learning to work with dynamic patterns, namely speech utterances, and perform a first experimental demonstration that it prevents acoustic data leakage in ASR training. We evaluate three end-to-end deep models, including LAS, hybrid CTC/attention, and RNN transducer, on the open-source LibriSpeech and TIMIT corpora. PATE learning-enhanced ASR models outperform the benchmark DP-SGD mechanisms, especially under strict DP budgets, giving relative word error rate reductions between 26.2% and 27.5% for an RNN transducer model evaluated with LibriSpeech. We also introduce a DP-preserving ASR solution for pretraining on public speech corpora.