Super-convergence and Differential Privacy: Training faster with better privacy guarantees

TL;DR

This paper demonstrates that super-convergence significantly accelerates training of differentially private neural networks, leading to faster training, higher accuracy, and improved privacy guarantees compared to traditional methods.

Contribution

It introduces the application of super-convergence to differentially private neural networks, achieving faster training and better privacy-utility trade-offs.

Findings

Super-convergence accelerates private neural network training by orders of magnitude.

Combining super-convergence with differential privacy improves validation accuracy.

Super-convergence enhances privacy guarantees in neural network training.

Abstract

The combination of deep neural networks and Differential Privacy has been of increasing interest in recent years, as it offers important data protection guarantees to the individuals of the training datasets used. However, using Differential Privacy in the training of neural networks comes with a set of shortcomings, like a decrease in validation accuracy and a significant increase in the use of resources and time in training. In this paper, we examine super-convergence as a way of greatly increasing training speed of differentially private neural networks, addressing the shortcoming of high training time and resource use. Super-convergence allows for acceleration in network training using very high learning rates, and has been shown to achieve models with high utility in orders of magnitude less training iterations than conventional ways. Experiments in this paper show that this order-of-magnitude speedup can also be seen when combining it with Differential Privacy, allowing for higher validation accuracies in much fewer training iterations compared to non-private, non-super convergent baseline models. Furthermore, super-convergence is shown to improve the privacy guarantees of private models.