A methodology for training homomorphicencryption friendly neural networks

TL;DR

This paper presents a methodology for training neural networks compatible with homomorphic encryption by replacing ReLU with quadratic activations, using transfer learning and knowledge distillation to maintain accuracy in privacy-preserving inference.

Contribution

The authors introduce a structured approach to develop HE-friendly neural networks, including activation function replacement and training techniques to preserve accuracy.

Findings

Reduced accuracy gap to within 0.32-5.3% using the proposed method.

Achieved 7% accuracy and F1 improvements over other HE-friendly training methods.

Demonstrated effectiveness on AlexNet and SqueezeNet architectures for COVID-19 detection.

Abstract

Privacy-preserving deep neural network (DNN) inference is a necessity in different regulated industries such as healthcare, finance and retail. Recently, homomorphic encryption (HE) has been used as a method to enable analytics while addressing privacy concerns. HE enables secure predictions over encrypted data. However, there are several challenges related to the use of HE, including DNN size limitations and the lack of support for some operation types. Most notably, the commonly used ReLU activation is not supported under some HE schemes. We propose a structured methodology to replace ReLU with a quadratic polynomial activation. To address the accuracy degradation issue, we use a pre-trained model that trains another HE-friendly model, using techniques such as trainable activation functions and knowledge distillation. We demonstrate our methodology on the AlexNet architecture, using the chest X-Ray and CT datasets for COVID-19 detection. Experiments using our approach reduced the gap between the F1 score and accuracy of the models trained with ReLU and the HE-friendly model to within a mere 0.32-5.3 percent degradation. We also demonstrate our methodology using the SqueezeNet architecture, for which we observed 7 percent accuracy and F1 improvements over training similar networks with other HE-friendly training methods.