PrivDFS: Private Inference via Distributed Feature Sharing against Data Reconstruction Attacks

TL;DR

PrivDFS introduces a distributed feature-sharing framework that fragments intermediate representations in image classification models, significantly reducing the success of data reconstruction attacks while maintaining high task accuracy.

Contribution

It proposes a novel, architecture-agnostic method of feature partitioning with learnable masks to enhance privacy in cloud-based vision inference.

Findings

DRA PSNR drops from 23.25 to 12.72 on CIFAR-10

Maintains within 1% of non-private accuracy

Effective across CNNs and Vision Transformers

Abstract

In this paper, we introduce PrivDFS, a distributed feature-sharing framework for input-private inference in image classification. A single holistic intermediate representation in split inference gives diffusion-based Data Reconstruction Attacks (DRAs) sufficient signal to reconstruct the input with high fidelity. PrivDFS restructures this vulnerability by fragmenting the representation and processing the fragments independently across a majority-honest set of servers. As a result, each branch observes only an incomplete and reconstruction-insufficient view of the input. To realize this, PrivDFS employs learnable binary masks that partition the intermediate representation into sparse and largely non-overlapping feature shares, each processed by a separate server, while a lightweight fusion module aggregates their predictions on the client. This design preserves full task accuracy when all branches are combined, yet sharply limits the reconstructive power available to any individual server. PrivDFS applies seamlessly to both ResNet-based CNNs and Vision Transformers. Across CIFAR-10/100, CelebA, and ImageNet-1K, PrivDFS induces a pronounced collapse in DRA performance, e.g., on CIFAR-10, PSNR drops from 23.25 -> 12.72 and SSIM from 0.963 -> 0.260, while maintaining accuracy within 1% of non-private split inference. These results establish structural feature partitioning as a practical and architecture-agnostic approach to reducing reconstructive leakage in cloud-based vision inference.