Network Moments: Extensions and Sparse-Smooth Attacks

TL;DR

This paper derives exact formulas for the mean and variance of small piecewise linear neural networks under Gaussian inputs, enabling the creation of sparse, smooth adversarial attacks and improving robustness analysis.

Contribution

It generalizes second-moment expressions for PL networks to arbitrary Gaussian inputs and demonstrates their use in constructing perceptually feasible adversarial attacks.

Findings

Derived tight variance estimates for PL networks.

Validated moment expressions through experiments on deeper networks.

Constructed effective sparse and smooth Gaussian adversarial attacks.

Abstract

The impressive performance of deep neural networks (DNNs) has immensely strengthened the line of research that aims at theoretically analyzing their effectiveness. This has incited research on the reaction of DNNs to noisy input, namely developing adversarial input attacks and strategies that lead to robust DNNs to these attacks. To that end, in this paper, we derive exact analytic expressions for the first and second moments (mean and variance) of a small piecewise linear (PL) network (Affine, ReLU, Affine) subject to Gaussian input. In particular, we generalize the second-moment expression of Bibi et al. to arbitrary input Gaussian distributions, dropping the zero-mean assumption. We show that the new variance expression can be efficiently approximated leading to much tighter variance estimates as compared to the preliminary results of Bibi et al. Moreover, we experimentally show that these expressions are tight under simple linearizations of deeper PL-DNNs, where we investigate the effect of the linearization sensitivity on the accuracy of the moment estimates. Lastly, we show that the derived expressions can be used to construct sparse and smooth Gaussian adversarial attacks (targeted and non-targeted) that tend to lead to perceptually feasible input attacks.