SoK: Privacy-preserving Deep Learning with Homomorphic Encryption

TL;DR

This paper systematically reviews how homomorphic encryption can be integrated with neural networks to enable privacy-preserving deep learning, highlighting the challenges and architectural modifications involved.

Contribution

It provides a comprehensive categorization of neural network adaptations for homomorphic encryption and analyzes their impact on performance and usability.

Findings

Numerous challenges include computational overhead and usability issues.

Encryption schemes impose limitations on neural network architectures.

Significant modifications are required for neural networks to operate over encrypted data.

Abstract

Outsourced computation for neural networks allows users access to state of the art models without needing to invest in specialized hardware and know-how. The problem is that the users lose control over potentially privacy sensitive data. With homomorphic encryption (HE) computation can be performed on encrypted data without revealing its content. In this systematization of knowledge, we take an in-depth look at approaches that combine neural networks with HE for privacy preservation. We categorize the changes to neural network models and architectures to make them computable over HE and how these changes impact performance. We find numerous challenges to HE based privacy-preserving deep learning such as computational overhead, usability, and limitations posed by the encryption schemes.