Inhibitor Transformers and Gated RNNs for Torus Efficient Fully Homomorphic Encryption

TL;DR

This paper proposes inhibitor-based modifications to Transformers and Gated RNNs that replace costly multiplications with additive operations, enabling efficient encrypted inference under Fully Homomorphic Encryption while maintaining competitive accuracy.

Contribution

It introduces inhibitor designs that enable integer-only, low-depth neural network architectures suitable for FHE, significantly improving encrypted inference efficiency.

Findings

3-6 times speedup in encrypted inference

30-50% reduction in plaintext inference time

Maintains competitive accuracy on benchmark datasets

Abstract

This paper introduces efficient modifications to neural network-based sequence processing approaches, laying new grounds for scalable privacy-preserving machine learning under Fully Homomorphic Encryption (FHE). Transformers are now ubiquitous in AI applications and have largely supplanted Gated Recurrent Neural Networks (RNNs) as the standard architecture for sequence modeling. Both architectures rely on costly multiplications and complex activations that hinder encrypted inference. We focus on TFHE, which supports deep circuit evaluation and efficient univariate function evaluation but makes variable-to-variable multiplication particularly expensive. To address this, we propose inhibitor designs for Transformers and gated RNNs that replace multiplications and Softmax/Sigmoid activations with additive and ReLU-based operations. These changes enable integer-only computation, reduce circuit depth, and improve the efficiency of encrypted execution while preserving learning capacity. We present complexity analyses and scaling experiments that indicate significant reductions in circuit depth and execution time under TFHE, with 3-6 times speedup for encrypted inference and 30-50% reductions in plaintext inference time. Empirical evaluations on MNIST, IMDB, and IAM handwriting show inhibitor-based models maintain competitive accuracy. Knowledge distillation further demonstrates that an inhibitor-based DistilBERT achieves performance close to that of the conventional attention model on GLUE, positioning these architectures as a viable approach for scalable, privacy-preserving AI.