Privacy Preserving Properties of Vision Classifiers

TL;DR

This paper evaluates how different vision classifier architectures like MLPs, CNNs, and ViTs vary in their vulnerability to inversion attacks, revealing privacy risks and guiding design choices for secure models.

Contribution

It systematically compares privacy-preserving properties of diverse vision architectures against inversion attacks, highlighting architectural factors influencing privacy risks.

Findings

CNNs and ViTs are more vulnerable to inversion attacks than MLPs.

Architectural features like input representation affect privacy risks.

Trade-offs exist between model accuracy and privacy protection.

Abstract

Vision classifiers are often trained on proprietary datasets containing sensitive information, yet the models themselves are frequently shared openly under the privacy-preserving assumption. Although these models are assumed to protect sensitive information in their training data, the extent to which this assumption holds for different architectures remains unexplored. This assumption is challenged by inversion attacks which attempt to reconstruct training data from model weights, exposing significant privacy vulnerabilities. In this study, we systematically evaluate the privacy-preserving properties of vision classifiers across diverse architectures, including Multi-Layer Perceptrons (MLPs), Convolutional Neural Networks (CNNs), and Vision Transformers (ViTs). Using network inversion-based reconstruction techniques, we assess the extent to which these architectures memorize and reveal training data, quantifying the relative ease of reconstruction across models. Our analysis highlights how architectural differences, such as input representation, feature extraction mechanisms, and weight structures, influence privacy risks. By comparing these architectures, we identify which are more resilient to inversion attacks and examine the trade-offs between model performance and privacy preservation, contributing to the development of secure and privacy-respecting machine learning models for sensitive applications. Our findings provide actionable insights into the design of secure and privacy-aware machine learning systems, emphasizing the importance of evaluating architectural decisions in sensitive applications involving proprietary or personal data.