Optimized Layerwise Approximation for Efficient Private Inference on Fully Homomorphic Encryption

TL;DR

This paper introduces an optimized layerwise approximation framework for private inference with homomorphic encryption, significantly reducing inference time while maintaining accuracy by customizing polynomial degrees per layer.

Contribution

It proposes a systematic, layerwise polynomial approximation method with a dynamic programming solution for efficient private inference without retraining.

Findings

Reduces inference time for ResNet models by over 2.8 times.

Successfully classifies CIFAR-10 with low-degree polynomials replacing GELU.

Achieves high accuracy without retraining by layerwise optimization.

Abstract

Recent studies have explored the deployment of privacy-preserving deep neural networks utilizing homomorphic encryption (HE), especially for private inference (PI). Many works have attempted the approximation-aware training (AAT) approach in PI, changing the activation functions of a model to low-degree polynomials that are easier to compute on HE by allowing model retraining. However, due to constraints in the training environment, it is often necessary to consider post-training approximation (PTA), using the pre-trained parameters of the existing plaintext model without retraining. Existing PTA studies have uniformly approximated the activation function in all layers to a high degree to mitigate accuracy loss from approximation, leading to significant time consumption. This study proposes an optimized layerwise approximation (OLA), a systematic framework that optimizes both accuracy loss and time consumption by using different approximation polynomials for each layer in the PTA scenario. For efficient approximation, we reflect the layerwise impact on the classification accuracy by considering the actual input distribution of each activation function while constructing the optimization problem. Additionally, we provide a dynamic programming technique to solve the optimization problem and achieve the optimized layerwise degrees in polynomial time. As a result, the OLA method reduces inference times for the ResNet-20 model and the ResNet-32 model by 3.02 times and 2.82 times, respectively, compared to prior state-of-the-art implementations employing uniform degree polynomials. Furthermore, we successfully classified CIFAR-10 by replacing the GELU function in the ConvNeXt model with only 3-degree polynomials using the proposed method, without modifying the backbone model.