Privacy-Preserving Vision Transformer Using Images Encrypted with Restricted Random Permutation Matrices

TL;DR

This paper introduces a new encryption technique for images using restricted random permutation matrices that enables privacy-preserving fine-tuning of vision transformers with better performance than traditional encryption methods.

Contribution

The paper presents a novel image encryption method with restricted random permutation matrices that maintains higher model accuracy during privacy-preserving vision transformer training.

Findings

Encryption with restricted random permutation matrices outperforms conventional methods in model accuracy.

The proposed method effectively balances privacy and performance.

Fine-tuning vision transformers on encrypted images is feasible with the new encryption scheme.

Abstract

We propose a novel method for privacy-preserving fine-tuning vision transformers (ViTs) with encrypted images. Conventional methods using encrypted images degrade model performance compared with that of using plain images due to the influence of image encryption. In contrast, the proposed encryption method using restricted random permutation matrices can provide a higher performance than the conventional ones.