PrivacyBench: Privacy Isn't Free in Hybrid Privacy-Preserving Vision Systems

TL;DR

PrivacyBench is a benchmarking framework that systematically evaluates hybrid privacy-preserving techniques in vision systems, revealing critical interactions and guiding robust deployment in sensitive applications.

Contribution

We introduce PrivacyBench, the first systematic platform for evaluating privacy-utility-cost trade-offs in hybrid privacy-preserving vision systems.

Findings

FL + DP shows severe convergence failure and accuracy drop

FL + SMPC maintains near-baseline performance with modest overhead

Privacy techniques cannot be combined arbitrarily, affecting deployment decisions

Abstract

Privacy preserving machine learning deployments in sensitive deep learning applications; from medical imaging to autonomous systems; increasingly require combining multiple techniques. Yet, practitioners lack systematic guidance to assess the synergistic and non-additive interactions of these hybrid configurations, relying instead on isolated technique analysis that misses critical system level interactions. We introduce PrivacyBench, a benchmarking framework that reveals striking failures in privacy technique combinations with severe deployment implications. Through systematic evaluation across ResNet18 and ViT models on medical datasets, we uncover that FL + DP combinations exhibit severe convergence failure, with accuracy dropping from 98% to 13% while compute costs and energy consumption substantially increase. In contrast, FL + SMPC maintains near-baseline performance with modest overhead. Our framework provides the first systematic platform for evaluating privacy-utility-cost trade-offs through automated YAML configuration, resource monitoring, and reproducible experimental protocols. PrivacyBench enables practitioners to identify problematic technique interactions before deployment, moving privacy-preserving computer vision from ad-hoc evaluation toward principled systems design. These findings demonstrate that privacy techniques cannot be composed arbitrarily and provide critical guidance for robust deployment in resource-constrained environments.