On the Vulnerability of Skip Connections to Model Inversion Attacks

TL;DR

This paper reveals that skip connections in deep neural networks increase vulnerability to model inversion attacks, especially in the last stages, and proposes new architectures that improve privacy protection against such attacks.

Contribution

It is the first study to analyze how skip connections affect model inversion attacks and introduces MI-resilient architectures that outperform existing defenses.

Findings

Skip connections reinforce model inversion attacks.

Skip connections in the last stage are most critical for attacks.

Proposed MI-resilient architectures outperform state-of-the-art defenses.

Abstract

Skip connections are fundamental architecture designs for modern deep neural networks (DNNs) such as CNNs and ViTs. While they help improve model performance significantly, we identify a vulnerability associated with skip connections to Model Inversion (MI) attacks, a type of privacy attack that aims to reconstruct private training data through abusive exploitation of a model. In this paper, as a pioneer work to understand how DNN architectures affect MI, we study the impact of skip connections on MI. We make the following discoveries: 1) Skip connections reinforce MI attacks and compromise data privacy. 2) Skip connections in the last stage are the most critical to attack. 3) RepVGG, an approach to remove skip connections in the inference-time architectures, could not mitigate the vulnerability to MI attacks. 4) Based on our findings, we propose MI-resilient architecture designs for the first time. Without bells and whistles, we show in extensive experiments that our MI-resilient architectures can outperform state-of-the-art (SOTA) defense methods in MI robustness. Furthermore, our MI-resilient architectures are complementary to existing MI defense methods. Our project is available at https://Pillowkoh.github.io/projects/RoLSS/