Modeling Neural Networks with Privacy Using Neural Stochastic Differential Equations

TL;DR

This paper explores the use of neural stochastic differential equations (NSDEs) to model neural networks with inherent privacy features, demonstrating their resistance to membership inference attacks and their potential as differentially-private learners.

Contribution

The paper introduces NSDEs as a novel approach to enhance privacy in neural networks, providing theoretical and empirical evidence of their differential privacy guarantees and attack resistance.

Findings

NSDEs exhibit twice the resistance to membership inference attacks compared to ResNets.

Limited expressivity of NODEs reduces overfitting and privacy risks.

NSDEs achieve privacy guarantees comparable to DP-SGD with better privacy-utility trade-offs.

Abstract

In this work, we study the feasibility of using neural ordinary differential equations (NODEs) to model systems with intrinsic privacy properties. Unlike conventional feedforward neural networks, which have unlimited expressivity and can represent arbitrary mappings between inputs and outputs, NODEs constrain their learning to the solution of a system of differential equations. We first examine whether this constraint reduces memorization and, consequently, the membership inference risks associated with NODEs. We conduct a comprehensive evaluation of NODEs under membership inference attacks and show that they exhibit twice the resistance compared to conventional models such as ResNets. By analyzing the variance in membership risks across different NODE models, we find that their limited expressivity leads to reduced overfitting to the training data. We then demonstrate, both theoretically and empirically, that membership inference risks can be further mitigated by utilizing a stochastic variant of NODEs: neural stochastic differential equations (NSDEs). We show that NSDEs are differentially-private (DP) learners that provide the same provable privacy guarantees as DPSGD, the de-facto mechanism for training private models. NSDEs are also effective in mitigating membership inference attacks, achieving risk levels comparable to private models trained with DP-SGD while offering an improved privacyutility trade-off. Moreover, we propose a drop-in-replacement strategy that efficiently integrates NSDEs into conventional feedforward architectures to enhance their privacy.