NeuJeans: Private Neural Network Inference with Joint Optimization of Convolution and FHE Bootstrapping

TL;DR

NeuJeans introduces a novel encoding and optimized execution flow for FHE-based private neural network inference, significantly reducing computation time and enabling CNN inference on large datasets like ImageNet within seconds.

Contribution

NeuJeans proposes a new CinS encoding method and optimized convolution execution flows, advancing the efficiency of FHE-based private CNN inference.

Findings

Achieves up to 5.68x speedup over previous FHE PI methods.

Enables CNN inference on ImageNet dataset within seconds.

Introduces a novel encoding that reduces costly permutations in FHE computations.

Abstract

Fully homomorphic encryption (FHE) is a promising cryptographic primitive for realizing private neural network inference (PI) services by allowing a client to fully offload the inference task to a cloud server while keeping the client data oblivious to the server. This work proposes NeuJeans, an FHE-based solution for the PI of deep convolutional neural networks (CNNs). NeuJeans tackles the critical problem of the enormous computational cost for the FHE evaluation of CNNs. We introduce a novel encoding method called Coefficients-in-Slot (CinS) encoding, which enables multiple convolutions in one HE multiplication without costly slot permutations. We further observe that CinS encoding is obtained by conducting the first several steps of the Discrete Fourier Transform (DFT) on a ciphertext in conventional Slot encoding. This property enables us to save the conversion between CinS and Slot encodings as bootstrapping a ciphertext starts with DFT. Exploiting this, we devise optimized execution flows for various two-dimensional convolution (conv2d) operations and apply them to end-to-end CNN implementations. NeuJeans accelerates the performance of conv2d-activation sequences by up to 5.68 times compared to state-of-the-art FHE-based PI work and performs the PI of a CNN at the scale of ImageNet within a mere few seconds.