A Systematic Comparison of Encrypted Machine Learning Solutions for Image Classification

TL;DR

This paper systematically compares various privacy-preserving machine learning frameworks for image classification, analyzing their performance, usability, and accuracy through experiments on standard datasets.

Contribution

It provides a comprehensive evaluation of four state-of-the-art secure computing frameworks for private image classification, highlighting their practical performance and usability.

Findings

TF-Trusted and CrypTen showed satisfying performance

All frameworks preserved model accuracy

Evaluation on MNIST and Malaria datasets

Abstract

This work provides a comprehensive review of existing frameworks based on secure computing techniques in the context of private image classification. The in-depth analysis of these approaches is followed by careful examination of their performance costs, in particular runtime and communication overhead. To further illustrate the practical considerations when using different privacy-preserving technologies, experiments were conducted using four state-of-the-art libraries implementing secure computing at the heart of the data science stack: PySyft and CrypTen supporting private inference via Secure Multi-Party Computation, TF-Trusted utilising Trusted Execution Environments and HE- Transformer relying on Homomorphic encryption. Our work aims to evaluate the suitability of these frameworks from a usability, runtime requirements and accuracy point of view. In order to better understand the gap between state-of-the-art protocols and what is currently available in practice for a data scientist, we designed three neural network architecture to obtain secure predictions via each of the four aforementioned frameworks. Two networks were evaluated on the MNIST dataset and one on the Malaria Cell image dataset. We observed satisfying performances for TF-Trusted and CrypTen and noted that all frameworks perfectly preserved the accuracy of the corresponding plaintext model.