A Distributed Framework for Privacy-Enhanced Vision Transformers on the Edge

TL;DR

This paper introduces a distributed, privacy-preserving framework for Vision Transformers that partitions visual data across multiple cloud servers, preventing complete data reconstruction and enhancing privacy while maintaining high performance.

Contribution

It proposes a novel hierarchical offloading framework using an edge orchestrator to improve privacy in vision tasks without sacrificing accuracy.

Findings

Significantly reduces data reconstruction risk compared to traditional cloud methods.

Maintains near-baseline segmentation performance with enhanced privacy.

Demonstrates scalability and practicality on real-world vision models like SAM.

Abstract

Nowadays, visual intelligence tools have become ubiquitous, offering all kinds of convenience and possibilities. However, these tools have high computational requirements that exceed the capabilities of resource-constrained mobile and wearable devices. While offloading visual data to the cloud is a common solution, it introduces significant privacy vulnerabilities during transmission and server-side computation. To address this, we propose a novel distributed, hierarchical offloading framework for Vision Transformers (ViTs) that addresses these privacy challenges by design. Our approach uses a local trusted edge device, such as a mobile phone or an Nvidia Jetson, as the edge orchestrator. This orchestrator partitions the user's visual data into smaller portions and distributes them across multiple independent cloud servers. By design, no single external server possesses the complete image, preventing comprehensive data reconstruction. The final data merging and aggregation computation occurs exclusively on the user's trusted edge device. We apply our framework to the Segment Anything Model (SAM) as a practical case study, which demonstrates that our method substantially enhances content privacy over traditional cloud-based approaches. Evaluations show our framework maintains near-baseline segmentation performance while substantially reducing the risk of content reconstruction and user data exposure. Our framework provides a scalable, privacy-preserving solution for vision tasks in the edge-cloud continuum.