CryptPEFT: Efficient and Private Neural Network Inference via Parameter-Efficient Fine-Tuning

TL;DR

CryptPEFT introduces a novel one-way communication architecture for private neural network inference, significantly reducing computational overhead while maintaining high accuracy, thus enabling efficient privacy-preserving machine learning.

Contribution

It proposes CryptPEFT, the first PEFT solution with one-way communication for private inference, optimizing adapter design via automated search for efficiency and utility.

Findings

Achieves 20.62x to 291.48x speedup over baselines.

Attains 85.47% accuracy on CIFAR-100.

Reduces encrypted computation to only the adapter.

Abstract

Publicly available large pretrained models (i.e., backbones) and lightweight adapters for parameter-efficient fine-tuning (PEFT) have become standard components in modern machine learning pipelines. However, preserving the privacy of both user inputs and fine-tuned adapters -- often trained on sensitive data -- during inference remains a significant challenge. Applying cryptographic techniques, such as multi-party computation (MPC), to PEFT settings still incurs substantial encrypted computation across both the backbone and adapter, mainly due to the inherent two-way communication between them. To address this limitation, we propose CryptPEFT, the first PEFT solution specifically designed for private inference scenarios. CryptPEFT introduces a novel one-way communication (OWC) architecture that confines encrypted computation solely to the adapter, significantly reducing both computational and communication overhead. To maintain strong model utility under this constraint, we explore the design space of OWC-compatible adapters and employ an automated architecture search algorithm to optimize the trade-off between private inference efficiency and model utility. We evaluated CryptPEFT using Vision Transformer backbones across widely used image classification datasets. Our results show that CryptPEFT significantly outperforms existing baselines, delivering speedups ranging from $20.62\times$ to $291.48\times$ in simulated wide-area network (WAN) and local-area network (LAN) settings. On CIFAR-100, CryptPEFT attains 85.47% accuracy with just 2.26 seconds of inference latency. These findings demonstrate that CryptPEFT offers an efficient and privacy-preserving solution for modern PEFT-based inference.