Robust Learning via Ensemble Density Propagation in Deep Neural Networks

TL;DR

This paper introduces Ensemble Density Propagation (EnDP), a Bayesian-based method that enhances the robustness of deep neural networks against noise and adversarial attacks by propagating distribution moments through layers.

Contribution

It presents a novel EnDP scheme for density propagation in Bayesian DNNs, improving robustness in uncertain environments with a theoretically grounded approach.

Findings

Significant robustness improvements on MNIST and CIFAR-10 datasets.

Effective propagation of distribution moments across network layers.

Enhanced resistance to adversarial attacks and noise.

Abstract

Learning in uncertain, noisy, or adversarial environments is a challenging task for deep neural networks (DNNs). We propose a new theoretically grounded and efficient approach for robust learning that builds upon Bayesian estimation and Variational Inference. We formulate the problem of density propagation through layers of a DNN and solve it using an Ensemble Density Propagation (EnDP) scheme. The EnDP approach allows us to propagate moments of the variational probability distribution across the layers of a Bayesian DNN, enabling the estimation of the mean and covariance of the predictive distribution at the output of the model. Our experiments using MNIST and CIFAR-10 datasets show a significant improvement in the robustness of the trained models to random noise and adversarial attacks.