Architecture Matters: Investigating the Influence of Differential Privacy on Neural Network Design

TL;DR

This paper explores how neural network architecture choices impact model accuracy under differential privacy constraints, revealing that traditional design principles do not directly transfer to privacy-preserving settings.

Contribution

It demonstrates that existing neural network architecture knowledge does not apply straightforwardly in differentially private contexts, highlighting the need for new design strategies.

Findings

Architectures effective without privacy constraints may underperform with differential privacy.

Current neural network design principles do not seamlessly translate to privacy-preserving models.

Further research is needed to optimize architectures for differential privacy.

Abstract

One barrier to more widespread adoption of differentially private neural networks is the entailed accuracy loss. To address this issue, the relationship between neural network architectures and model accuracy under differential privacy constraints needs to be better understood. As a first step, we test whether extant knowledge on architecture design also holds in the differentially private setting. Our findings show that it does not; architectures that perform well without differential privacy, do not necessarily do so with differential privacy. Consequently, extant knowledge on neural network architecture design cannot be seamlessly translated into the differential privacy context. Future research is required to better understand the relationship between neural network architectures and model accuracy to enable better architecture design choices under differential privacy constraints.