Minimum Uncertainty Based Detection of Adversaries in Deep Neural Networks

TL;DR

This paper introduces a modular, minimum uncertainty-based randomized detection method for adversarial attacks on deep neural networks, leveraging Bayesian-inspired sampling at hidden layers without retraining.

Contribution

It proposes a novel, scalable convex optimization approach for selecting sampling probabilities to detect adversarial inputs effectively in pre-trained DNNs.

Findings

Effective detection of various adversarial attacks demonstrated

Method achieves high accuracy with no additional training overhead

Links to state-of-the-art randomized adversarial detectors

Abstract

Despite their unprecedented performance in various domains, utilization of Deep Neural Networks (DNNs) in safety-critical environments is severely limited in the presence of even small adversarial perturbations. The present work develops a randomized approach to detecting such perturbations based on minimum uncertainty metrics that rely on sampling at the hidden layers during the DNN inference stage. Inspired by Bayesian approaches to uncertainty estimation, the sampling probabilities are designed for effective detection of the adversarially corrupted inputs. Being modular, the novel detector of adversaries can be conveniently employed by any pre-trained DNN at no extra training overhead. Selecting which units to sample per hidden layer entails quantifying the amount of DNN output uncertainty, where the overall uncertainty is expressed in terms of its layer-wise components - what also promotes scalability. Sampling probabilities are then sought by minimizing uncertainty measures layer-by-layer, leading to a novel convex optimization problem that admits an exact solver with superlinear convergence rate. By simplifying the objective function, low-complexity approximate solvers are also developed. In addition to valuable insights, these approximations link the novel approach with state-of-the-art randomized adversarial detectors. The effectiveness of the novel detectors in the context of competing alternatives is highlighted through extensive tests for various types of adversarial attacks with variable levels of strength.

Tables

Table 1. TABLE I: List of variables

Param.	Definition
${𝐱}_{\nu }$	Test input image
${𝐲}_{\nu }^{\text{target}}$	Ground-truth one-hot class label for input ${𝐱}_{\nu }$
${𝐲}_{\nu }$	Output for ${𝐱}_{\nu }$ by the deterministic classifier
${𝐲}_{\nu }^{\left(r\right)}$	Output for ${𝐱}_{\nu }$ in the $r$-th realization
	of the detection network (with sampling units)
${𝐱}_{\left(l\right)}$	Value at hidden layer $l$ with width $h_l$
${𝐖}_l$	Network weight at layer $l$ (random variable)
${𝐖}_l^{\text{TR}}$	Network weights given by the training phase
${𝐌}_l$	Expected value of ${𝐖}_l$
${𝐒}_l$	Sampling matrix, defined as ${𝐒}_l=\text{diag}\left({𝐳}_l\right)$
${𝐃}_l$	Diagonal matrix ${𝐃}_l={\text{diag}}^{†}\left({𝔼}_{q_z({𝐳}_l;{𝐩}_l)}\left[{𝐳}_l\right]\right)$
${𝐳}_l$	$h_l$-dimensional binary multivariate, modeling
	the overall sampling outcome at layer $l$
${𝜻}_l^c$	Categorical $h_l$-dimensional binary multivariate,
	modeling the $c$-th draw at layer $l$
${𝐩}_l$	Parameters of the categorical pmf of ${𝜻}_l^{\left(c\right)}$
$C$	Total number of draws
$f$	Scaler coeff. in $[0,1]$ defining $C=f\times \text{nnz}\left({𝐱}_l\right)$
$B$	Resnet architecture blocks in Tables 3 and 4

Table 2. TABLE II: CIFAR10 and cats-and-dogs image-classification datasets

Dataset	image size	# train	# val.	# test
CIFAR10	32 x 32	50,000	2,000	8,000
Cats-and-dogs	224 x 224	10,000	2,000	13,000

Table 3. TABLE III: ResNet20 architecture on CIFAR10 dataset

name	output-size	20 layers	#sampling units
Block1	32 x 32	[ 3 x 3, 16]	1
Block2	32 x 32	$\left[\begin{array}{c}3\times 3,16\\ 3\times 3,16\end{array}\right]\times 3$	6
Block3	16 x 16	$\left[\begin{array}{c}3\times 3,32\\ 3\times 3,32\end{array}\right]\times 3$	6
Block4	8 x 8	$\left[\begin{array}{c}3\times 3,64\\ 3\times 3,64\end{array}\right]\times 3$	6
		average pool, $64$-d fully conn., softmax
Block5	1 x 1		1

Table 4. TABLE X: AUC-ROC of different attack-detection sampling schemes on cats-and-dogs test set against BIM, C&W, and combination attacks. Higher values indicate better detection.

	BIM Attack				C&W Attack		Combination Attack
Sampling	$ϵ=10$		$ϵ=20$
Method	parameters	AUC	parameters	AUC	parameters	AUC	parameters	AUC
VM		71.4		70.7		71.6		70.4
VM-log	$(B,f)=(1,3.0)$	70.3	$(B,f)=(1,3.0)$	69.4	$(B,f)=(2,0.8)$	56.9	$(B,f)=(2,0.7)$	57.0
VM-linear		66.7		66.2		71.9		68.2
DVM-log	$(B,f)=(4,3.0)$	63.9	$(B,f)=(4,3.0)$	63.6	$(B,f)=(4,3.0)$	69.6	$(B,f)=(4,3.0)$	68.1
SAP	$(B,f)=(1,3.0)$	67.1	$(B,f)=(1,3.0)$	67.1	$(B,f)=(5,0.9)$	63.8	$(B,f)=(6,1.0)$	67.1
Dropout	$(B,{\pi }_{\text{drp}})=(5,0.6)$	60.1	$(B,{\pi }_{\text{drp}})=(6,0.1)$	59.6	$(B,{\pi }_{\text{drp}})=(6,0.5)$	63.1	$(B,{\pi }_{\text{drp}})=(6,0.1)$	66.1

Equations

$$p({\mathbf{y}}_{\nu }|{\mathbf{x}}_{\nu },\mathbf{X},\mathbf{Y})=\int p({\mathbf{y}}_{\nu }|{\mathbf{x}}_{\nu },\omega )p(\omega |\mathbf{X},\mathbf{Y})d\omega$$

$${\mathcal{L}}_{VI}(\theta ):=\int q_{\theta }(\omega )\log p(\mathbf{Y}|\mathbf{X},\omega )d\omega -KL(q_{\theta }(\omega )||p(\omega )).$$

$$q_{\theta }(\omega )=\prod _{l=1}^{L}q({\mathbf{W}}_l;{\mathbf{M}}_l,{\mathbf{\theta }}_{z_l})$$

$${\mathbf{W}}_l={\mathbf{M}}_l\text{diag}([z_{l,1},z_{l,2},\dots ,z_{l,h_l}]),l=1,\dots ,L$$

$$p({\mathbf{y}}_{\nu }|{\mathbf{x}}_{\nu },\mathbf{X},\mathbf{Y})\approx \int p({\mathbf{y}}_{\nu }|{\mathbf{x}}_{\nu },\omega )q_{\theta }(\omega )d\omega .$$

$$\mathbf{M}_{l}=\mathbf{W}^{(TR)}_{l}\;\text{diag}^{\dagger}\Big{(}\mathbb{E}_{q_{z}(\mathbf{z}_{l};\boldsymbol{\theta}_{z_{l}})}[\mathbf{z}_{l}]\Big{)}\>,\,\;\;l=1,\ldots,L$$

$${\mathbb{E}}_{q_{\theta }(\omega )}[{\mathbf{y}}_{\nu }|{\mathbf{x}}_{\nu };\{{\mathbf{\theta }}_{{\mathbf{z}}_l}{\}}_{l=1}^L]\simeq {\bar{\mathbf{y}}}_{\nu }=\frac{1}{R}\sum _{r=1}^R{\mathbf{y}}_{\nu }^{(r)}$$

$${\text{Cov}}_{q_{\theta }(\omega )}[{\mathbf{y}}_{\nu }|{\mathbf{x}}_{\nu };\{{\mathbf{\theta }}_{{\mathbf{z}}_l}{\}}_{l=1}^L]\simeq$$

$$\{\begin{array}{l}{\mathcal{H}}_0:{\mathbf{x}}_{\nu }={\mathbf{x}}_{\nu }^{\text{clean}}U({\mathbf{x}}_{\nu })\le {\tau }_0\\ {\mathcal{H}}_1:{\mathbf{x}}_{\nu }={\mathbf{x}}_{\nu }^{\text{clean}}+{\mathbf{n}}_{\nu }^{\text{adv}}U({\mathbf{x}}_{\nu })>{\tau }_0\end{array}$$

$${\mathbf{\zeta }}_l^{(c)}=[{\zeta }_{l,1}^{(c)},{\zeta }_{l,2}^{(c)}\dots {\zeta }_{l,h_l}^{(c)}{]}^{\top }\sim \text{Categorical}({\mathbf{p}}_l),c=1,\dots ,C$$

$${\zeta }_{l,i}^{(c)}=\{\begin{array}{l}1\text{if draw c picks the ith unit of hidden layer l}\\ 0\text{otherwise}\end{array}$$

$${\mathbf{z}}_l:={\mathbf{\zeta }}_l^{(1)}||{\mathbf{\zeta }}_l^{(2)}||...||{\mathbf{\zeta }}_l^{(C)}.$$

$$\{\begin{array}{l}{\mathcal{H}}_0:{\mathbf{x}}_{\nu }={\mathbf{x}}_{\nu }^{\text{clean}}{\min }_{\{{\mathbf{p}}_l{\}}_{\forall l}}U({\mathbf{x}}_{\nu };\{{\mathbf{p}}_l{\}}_{l=1}^L)\le {\tau }_0\\ {\mathcal{H}}_1:{\mathbf{x}}_{\nu }={\mathbf{x}}_{\nu }^{\text{clean}}+{\mathbf{n}}_{\nu }^{\text{adv}}\text{otherwise}\end{array}$$

$$\hat{I}({\mathbf{x}}_{\nu };\{{\mathbf{p}}_l{\}}_{l=1}^L):=H({\bar{\mathbf{y}}}_{\nu })-\frac{1}{R}\sum _{r=1}^{R}H({\mathbf{y}}_{\nu }^{(r)})$$

$$H({\mathbf{y}}_{\nu }):=-\sum _{k=1}^K{y}_{\nu ,k}\log (y_{\nu ,k}).$$

$$\underset{\{{\mathbf{p}}_l{\}}_{l=1}^L}{\min }\hat{I}({\mathbf{x}}_{\nu };\{{\mathbf{p}}_l{\}}_{l=1}^L)$$

$$\underset{\{{\mathbf{p}}_l{\}}_{l=1}^L}{\min }U({\mathbf{x}}_{\nu };\{{\mathbf{p}}_l{\}}_{l=1}^L)=\text{Tr}({\text{Cov}}_{q_{\theta }(\omega )}[{\mathbf{y}}_{\nu }]).$$

$$\mathbf{y}_{\nu}=\sigma_{\text{softmax}}\Big{(}\mathbf{W}_{2}\sigma(\mathbf{W}_{1}\mathbf{x}_{\nu})\Big{)}$$

$${\sigma }_{\text{softmax}}(\mathbf{u}):={\left[\frac{e^{u_1}}{{\sum }_{i=1}^K{e}^{u_i}},\frac{e^{u_2}}{{\sum }_{i=1}^K{e}^{u_i}},\dots ,\frac{e^{u_K}}{{\sum }_{i=1}^K{e}^{u_i}}\right]}^{\top }$$

$${\mathbf{W}}_l:={\mathbf{M}}_l\text{diag}({\mathbf{z}}_l)={\mathbf{W}}_l^{(TR)}{\mathbf{S}}_l{\mathbf{D}}_{l}l=1,2$$

$$\underset{\{{\mathbf{p}}_i\ge 0,1^{\top }{\mathbf{p}}_i=1{\}}_{i=1}^2}{\min }$$

$$+\gamma \text{Tr}({\mathbb{E}}_{{\mathbf{W}}_2}[{\mathbf{W}}_2{\mathbf{W}}_2^{\top }])\text{Tr}({\text{ Cov}}_{{\mathbf{W}}_1}[{\mathbf{W}}_1{\mathbf{x}}_{\nu }])$$

$$\text{Step 1:}{\mathbf{p}}_1^{\ast }=\arg \underset{{\mathbf{p}}_1}{\min }\text{Tr}({\text{ Cov}}_{{\mathbf{W}}_1}[{{\mathbf{W}}_1\mathbf{x}}_{\nu }])$$

$$\text{Step 2:}{\mathbf{p}}_2^{\ast }=\arg \underset{{\mathbf{p}}_2}{\min }\text{Tr}({\text{Cov}}_{{\mathbf{W}}_2}[{\mathbf{W}}_2\sigma ({\mathbf{W}}_1^{(TR)}{\mathbf{x}}_{\nu })])$$

$$+{\gamma }^{\prime }\text{Tr}({\mathbb{E}}_{{\mathbf{W}}_2}[{\mathbf{W}}_2{\mathbf{W}}_2^{\top }])$$

$${\mathbf{p}}_2^{\ast }=\arg \underset{{\mathbf{p}}_2}{\min }\text{Tr}({\text{Cov}}_{{\mathbf{W}}_2}[{\mathbf{W}}_2\sigma ({\mathbf{W}}_1^{(TR)}{\mathbf{x}}_{\nu })])$$

$${\mathbf{p}}_l^{\ast }=\arg \underset{{\mathbf{p}}_l}{\min }\text{Tr}({\text{ Cov}}_{{\mathbf{W}}_l}[{{\mathbf{W}}_l\mathbf{x}}_{(l-1)}])$$

$$({\text{ Cov}}_{{\mathbf{W}}_l}[{{\mathbf{W}}_l\mathbf{x}}_{(l-1)}])$$

$$\displaystyle=\mathbb{E}\Big{[}\|\mathbf{W}_{l}^{(TR)}\;\mathbf{S}_{l}\;\mathbf{D}_{l}\mathbf{x}_{(l-1)}-\mathbb{E}[\mathbf{W}_{l}^{(TR)}\;\mathbf{S}_{l}\;\mathbf{D}_{l}\mathbf{x}_{(l-1)}]\|_{2}^{2}\Big{]}$$

$$\displaystyle=\mathbb{E}\Big{[}\|\mathbf{W}_{l}^{(TR)}\;\mathbf{S}_{l}\;\mathbf{D}_{l}\mathbf{x}_{(l-1)}-\mathbf{W}_{l}^{(TR)}\;\mathbf{x}_{(l-1)}\|_{2}^{2}\Big{]}$$

$$\displaystyle\leq\|\mathbf{W}_{l}^{(TR)}\|_{2}^{2}\;\mathbb{E}\Big{[}\|\mathbf{S}_{l}\;\mathbf{D}_{l}\mathbf{x}_{(l-1)}-\mathbf{x}_{(l-1)}\|_{2}^{2}\Big{]}$$

$$=\Vert {\mathbf{W}}_l^{(TR)}{\Vert }_2^2\sum _{i=1}^{h_l}\mathbb{E}[(S_{ii}{D}_{ii}{\mathbf{x}}_{(l-1),i}-{\mathbf{x}}_{(l-1),i}{)}^2]$$

$$=\Vert {\mathbf{W}}_l^{(TR)}{\Vert }_2^2\sum _{i=1}^{h_l}{\mathbf{x}}_{(l-1),i}^2\mathbb{E}[(S_{ii}{D}_{ii}-1{)}^2]$$

$$=\Vert {\mathbf{W}}_l^{(TR)}{\Vert }_2^2\sum _{i=1}^{h_l}{\mathbf{x}}_{(l-1),i}^2\left(\frac{1}{{\pi }_{l,i}}-1\right)$$

Full text

1 Introduction

Unprecedented learning capability offered by Deep Neural Networks (DNNs) has enabled state-of-the-art performance in diverse tasks such as object recognition and detection [1, 2, 3], speech recognition and language translation [4], voice synthesis [5], and many more, to reach or even surpass human-level accuracy. Despite their performance however, recent studies have cast doubt on the reliability of DNNs as highly-accurate networks are shown to be extremely sensitive to carefully crafted inputs designed to fool them [6, 7, 8]. Such fragility can easily lead to sabotage once adversarial entities target critical environments such as autonomous cars [9], automatic speech recognition [10], and face detection [11, 12, 3]. The extreme brittleness of convolutional neural networks (CNNs) for image classification is highlighted since small adversarial perturbations on the clean image, although often imperceptible to the human eye, can lead the trained CNNs to classify the adversarial examples incorrectly with high confidence. In particular, design of powerful adversarial perturbations in environments with different levels of complexity and knowledge about the target CNN, known as white, grey, and black-box attacks, have been investigated in several works [7, 13, 14, 15, 16, 17]. These considerations motivate well the need for designing robust and powerful attack detection mechanisms for reliable and safe utilization of DNNs [18].

Defense against adversarial perturbations has been mainly pursued in two broad directions: (i) attack detection, and (ii) attack recovery. Methods in the first category aim at detecting adversarial corruption in the input by classifying the input as clean or adversarial, based on tools as diverse as auto-encoders [19], detection sub-networks [20, 21], and dropout units [22]. On the other hand, methods in the second category are based on recovery schemes that robustify the classification by data pre-processing and randomization [23, 24, 25], adversarial training [26, 27, 28, 29, 30], sparsification of the network [31, 32] and Lipschitz regularization [33, 34], to name just a few.

Furthermore, the so-termed over-confidence of DNNs in classifying “out-of-distribution,” meaning samples which lie in unexplored regions of the input domain, or even “misclassified” samples, has been unraveled in [35, 36]. This has motivated the need for uncertainty estimation as well as calibration of the networks for robust classification. Modern Bayesian neural networks target this issue by modeling the distribution of DNN weights as random [37], and estimating the DNN output uncertainty through predictive entropy, variance, or mutual information [38, 39, 22, 40]. The well-known dropout regularization technique is one such approximate Bayesian neural network, now widely used in training and testing of DNNs [41, 42, 43].

Moreover, approaches relying on dropout units have shown promising performance in successfully detecting adversarial attacks, where other defense mechanisms fail [13]. In particular, [22] utilizes randomness of dropout units during the test phase as a defense mechanism, and approximates the classification uncertainty by Monte Carlo (MC) estimation of the output variance. Based on the latter, images with high classification uncertainty are declared as adversarial. Recently, dropout defense has been generalized to non-uniform sampling [44], where entries of the hidden-layers are randomly sampled, with probabilities proportional to the entry values. This heuristic sampling of units per layer is inspired by intuitive reasoning: activation units with large entries have more information and should be sampled more often [44]. However, analytical understanding has not been investigated.

The goal here is to further expand the understanding of uncertainty estimation in DNNs, and thereby improve the detection of adversarial inputs. The premise is that inherent distance of the adversarial perturbation from the natural-image manifold will cause the overall network uncertainty to exceed that of the clean image, and thus successful detection can be obtained.

To this end, and inspired by [44], we rely on random sampling of units per hidden layer of a pre-trained network to introduce randomness. Moreover, inspired by the Bayesian approaches to uncertainty estimation, the overall uncertainty of a given image is then quantified in terms of its hidden-layer components. We then formulate the task of adversary detection as uncertainty minimization by optimizing over the sampling probabilities to provide effective detection. Subsequently, we develop an exact solver with super-linear convergence rate as well as approximate low-complexity solvers for an efficient layer-by-layer uncertainty minimization scheme. Furthermore, we draw connections with uniform dropout [22] as well as stochastic approximate pruning (SAP) [44], and provide an efficient implementation of the novel approach by interpreting it as a non-uniform dropout scheme. Extensive numerical tests on CIFAR10 and high-quality cats-and-dogs images in the presence of various attack schemes corroborate the importance of our designs of sampling probabilities, as well as the placement of sampling units per hidden layer for improved detection of adversarial inputs.

The rest of the paper is organized as follows. An overview on Bayesian inference and uncertainty-based detection in neural networks is provided in Section 2. Inspired by this, the proposed class of detectors is introduced in Section 3, and exact as well as low-complexity approximate solvers for the layer-by-layer uncertainty minimization are the subjects of Section 4. Implementation issues are dealt with in Section 5, numerical tests are provided in Section 6, and concluding remarks are discussed in Section 7.

2 Bayesian Neural Network Preliminaries

Bayesian inference is among the powerful tools utilized for analytically understanding and quantifying uncertainty in DNNs [45, 42]. In this section, we provide a short review on the basics of Bayesian neural networks, and move on to the inference phase for adversary detection in Section 2.2, which is of primary interest in this work.

Consider an $L$-layer deep neural network, which maps the input $\mathbf{x}\in\mathcal{X}$ to output $\mathbf{y}\in\mathcal{Y}$. The weights are denoted by $\omega:=\{\mathbf{W}_{l}\}_{l=1}^{L}$, and are modeled as random variables with prior probability density function (pdf) $p(\omega)$.

Given training input $\mathbf{X}:=[\mathbf{x}_{1},\mathbf{x}_{2},...,\mathbf{x}_{n}]$ and output data $\mathbf{Y}:=[\mathbf{y}_{1},\mathbf{y}_{2},...,\mathbf{y}_{n}]$, it is assumed that the parameters $\omega$ only depend on these $(\mathbf{X},\mathbf{Y})$ data. As a result, the predictive pdf for a new input $\mathbf{x}_{\nu}$ can be obtained via marginalization as [41]

[TABLE]

which requires knowing the conditional $p(\omega|\mathbf{X},\mathbf{Y})$. The complexity of estimating $p(\omega|\mathbf{X},\mathbf{Y})$ motivates well the variational inference (VI) approach, where $p(\omega|\mathbf{X},\mathbf{Y})$ is replaced by a surrogate pdf $q_{\theta}(\omega)$ that is parameterized by $\theta$. For $q_{\theta}(\omega)$, it is desired to: (D1) approximate closely $p(\omega|\mathbf{X},\mathbf{Y})$; and, (D2) provide easy marginalization in (1) either in closed form or empirically. To meet (D1), the surrogate is chosen by minimizing the Kullback-Leibler (KL) divergence $KL(p(\omega|\mathbf{X},\mathbf{Y}),q_{\theta}(\omega))$, which is subsequently approximated by the log evidence lower bound [46, p. 462]

[TABLE]

Finding $q_{\theta}$ boils down to maximizing the log evidence lower bound, that is, $\theta_{\text{VI}}=\arg\max_{\theta}\mathcal{L}_{\text{VI}}(\theta)$. A common choice for $q_{\theta}(\omega)$ to also satisfy (D2) is described next.

2.1 Variational inference

A simple yet effective choice for $q_{\theta}(\omega)$ is a factored form modeling the weights as independent across layers, that is

[TABLE]

where the $l$-th layer with $h_{l}$ hidden units is modeled as

[TABLE]

where $\mathbf{M}_{l}$ is an $h_{l+1}\times h_{l}$ deterministic weight matrix multiplied by a diagonal matrix formed by the binary random vector $\mathbf{z}_{l}:=[z_{l,1},z_{l,2},\ldots,z_{l,h_{l}}]\in\{0,1\}^{h_{l}}$ with entries drawn from a pmf $q_{z}(\mathbf{z}_{l};\boldsymbol{\theta}_{z_{l}})$ parameterized by $\boldsymbol{\theta}_{z_{l}}$.

If the entries $\{z_{l,i}\}$ are i.i.d. Bernoulli with (identical) probability (w.p.) $\pi$, they effect what is referred to as uniform (across layers and nodes) dropout, which is known to prevent overfitting [45]. Clearly, the parameter set ${\theta}:=\{\mathbf{M}_{l},\boldsymbol{\theta}_{z_{l}}\}_{l=1}^{L}=\{\mathbf{M}_{l}\}_{l=1}^{L}\cup\{\pi\}$ fully characterizes $q_{\theta}(\omega)$. The dropout probability $1-\pi$ is preselected in practice, while $\{\mathbf{M}_{l}\}_{l=1}^{L}$ can be obtained using the training data by maximizing the log evidence lower bound in (2). Nonetheless, integration in (2) over all the Bernoulli variables is analytically challenging, while sampling from the Bernoulli pmf is relatively cheap. This prompts approximate yet efficient integration using Monte Carlo estimation. A more detailed account of training Bayesian neural networks can be found in [41, 46, 47]. Moving on, the ensuing subsection deals with detection of adversarial inputs inspired by Bayesian neural networks.

2.2 Detection of DNN adversaries

In addition to facilitating ELBO maximization during the training phase, probabilistic view on the network parameters during the test phase can also be utilized towards output uncertainty estimation for the detection of adversarial inputs[41]. To do so, detection during the testing phase proceeds by approximating the predictive pdf in (1) using the variational surrogate $q_{\theta}(\omega)$, as

[TABLE]

Deciphering whether a given input $\mathbf{x}_{\nu}$ is adversarial entails three steps: (S1) parametric modeling of $q_{\theta}(\omega)$; (S2) estimating the DNN output uncertainty captured by $p(\mathbf{y}_{\nu}|\mathbf{x}_{\nu},\mathbf{X},\mathbf{Y})$; and (S3) declaring $\mathbf{x}_{\nu}$ as adversarial if the output uncertainty exceeds a certain threshold, and clean otherwise. These steps are elaborated next.

Step 1: Parametric modeling of $q_{\theta}{(\omega)}$. Recall that uniform dropout offers a popular special class of $q_{\theta}(\omega)$ pdfs, and has been employed in adversary detection [22]. Here, we specify the richer model of $q_{\theta}(\omega)$ in (3) and (4) that will turn out to markedly improve detection performance. Different from uniform dropout, we will allow for (possibly correlated) Bernoulli variables with carefully selected (possibly non-identical) parameters. If such general $\{\boldsymbol{\theta}_{z_{l}}\}_{l=1}^{L}$ can be obtained, matrices $\{\mathbf{M}_{l}\}_{l=1}^{L}$ are then found as follows.

Let $\{\mathbf{W}^{(TR)}_{l}\}_{l=1}^{L}$ be deterministic weight matrices obtained via non-Bayesian training that we denote as (TR)111Such as back propagation based on e.g., a cross-entropy criterion.. We will use $\mathbf{W}^{(TR)}_{l}$ to specify the mean of the random weight matrix $\mathbf{W}_{l}$ in our approach, meaning we choose $\mathbb{E}_{q_{z}(\mathbf{z}_{l};\boldsymbol{\theta}_{l})}[\mathbf{W}_{l}]\mathbf{x}_{(l-1)}=\mathbf{W}^{(TR)}_{l}\mathbf{x}_{(l-1)}$ $\forall l$, where $\mathbf{x}_{(l-1)}$ is the output of the $(l-1)$st layer for a given input $\mathbf{x}_{\nu}$ passing through the DNN with deterministic weights $\{\mathbf{W}^{(TR)}_{l}\}_{l=1}^{L}$. With $\mathbf{W}^{(TR)}_{l}$ available, we first design $q_{z}(\mathbf{z}_{l};\boldsymbol{\theta}_{z_{l}})$; next, we find $\boldsymbol{\theta}_{z_{l}}$; and then $\mathbf{M}_{l}$, as

[TABLE]

where the pseudo-inverse $\dagger$ means that inverse entries are replaced with zeros if $\mathbb{E}_{q_{z}(\mathbf{z}_{l};\boldsymbol{\theta}_{z_{l}})}[z_{l,i}]=0$.

Step 2: Quantifying the DNN output uncertainty. Since evaluation of $p(\mathbf{y}_{\nu}|\mathbf{x}_{\nu},\mathbf{X},\mathbf{Y})$ in (5) is prohibitive, one can estimate it using MC sampling. In particular, one can readily obtain MC estimates of (conditional) moments of $\mathbf{y}_{\nu}$. For instance, its mean and variance can be estimated as

[TABLE]

and

[TABLE]

where $\mathbf{y}_{\nu}^{(r)}$ is the output of the $r$-th DNN realized through weights $\{\mathbf{W}_{l}^{(r)}\}_{l=1}^{L}$ with input $\mathbf{x}_{\nu}$. The predictive variance is the trace of $\text{Cov}_{q_{\theta}(\omega)}[\mathbf{y}_{\nu}|\mathbf{x}_{\nu};\{\boldsymbol{\theta}_{\mathbf{z}_{l}}\}_{l=1}^{L}]$ that we henceforth abbreviate as $\text{Tr}(\text{Cov}_{q_{\theta}(\omega)}[\mathbf{y}_{\nu}|\mathbf{x}_{\nu}])$. Given $\mathbf{x}_{\nu}$, the latter has been used to quantify output uncertainty as $U(\mathbf{x}_{\nu})=\text{Tr}(\text{Cov}_{q_{\theta}(\omega)}[\mathbf{y}_{\nu}|\mathbf{x}_{\nu}])$ [22]. Additional measures of uncertainty will be presented in the next section.

Step 3: Detecting adversarial inputs. Given $U(\mathbf{x}_{\nu})$, detection of adversarial inputs is cast as testing the hypotheses

[TABLE]

where the null suggests absence of adversarial perturbation (low variance/uncertainty below threshold $\tau_{0}$), while the alternative in effect raises a red flag for presence of adversarial input (high variance/uncertainty above threshold $\tau_{0}$).

We will now proceed to introduce our novel variational distribution model targeting improved detection of adversaries based on uncertainty minimization.

3 Minimum Uncertainty based Detection

To design $q_{z}(\mathbf{z}_{l};\boldsymbol{\theta}_{z_{l}})$, we will build on and formalize the sampling scheme in [44] that is employed to specify the joint pmf of the (generally correlated) binary variables $\{z_{l,i}\}_{i=1}^{h_{l}}$ per layer $l$. To this end, we randomly pick one activation unit output of the $h_{l}$ hidden units per layer $l$; and repeat such a random draw $C$ times with replacement. Let $\boldsymbol{\zeta}_{l}^{(c)}$ denote per draw $c$ the $h_{l}\times 1$ vector variable

[TABLE]

where each entry $\zeta_{l,i}^{(c)}$ is a binary random variable with

[TABLE]

and the $h_{l}\times 1$ vector $\mathbf{p}_{l}$ with nonegative entries summing up to $1$ specifies the Categorical pmf of $\boldsymbol{\zeta}_{l}^{(c)}$.

With $||$ denoting element-wise binary OR operation on vectors $\{\boldsymbol{\zeta}_{l}^{(c)}\}_{c=1}^{C}$, we define next the vector

[TABLE]

Using $\mathbf{z}_{l}$ as in (9) with ${\{\boldsymbol{\theta}_{z_{l}}=\mathbf{p}_{l}\}_{l=1}^{L}}$ to be selected, enables finding the expectation and then $\mathbf{M}_{l}$ in (6). Deterministic matrix $\mathbf{M}_{l}$ along with the variates $\{\mathbf{z}_{l}^{(r)}\}_{r=1}^{R}$ provide the desired DNN realizations to estimate the uncertainty $U(\mathbf{x}_{\nu};{\{\mathbf{p}_{l}\}_{l=1}^{L}})=\text{Tr}(\text{Cov}_{q_{\theta}(\omega)}[\mathbf{y}_{\nu}|\mathbf{x}_{\nu}])$ as in (7). In turn, this leads to our novel adversarial input detector (cf. (8))

[TABLE]

where variational parameters ${\{\mathbf{p}_{l}\}_{l=1}^{L}}$ are sought such that uncertainty $U(\mathbf{x}_{\nu};\{\mathbf{p}_{l}\}_{l=1}^{L})$ is minimized under $\mathcal{H}_{0}$.

The rationale behind our detector in (10) is that for a given detection threshold $\tau_{0}$, uncertainty minimization will increase the number of clean images whose minimized uncertainty will fall below this threshold, and thus lead to a lower probability of false alarms. The probability of adversarial input detection however, depends on test statistic pdf under $\mathcal{H}_{1}$, in which the adversarial perturbation $\mathbf{n}_{\nu}^{\text{adv}}$ is unknown. The premise here is that due to network instability under $\mathcal{H}_{1}$, the sought probabilities ${\{\mathbf{p}_{l}\}_{l=1}^{L}}$ will not reduce uncertainty under $\mathcal{H}_{1}$ as effectively, thus minimum uncertainty-based detection can provide improved ROC curves. In lieu of analytical metrics, this has been tested through extensive numerical experiments, and its effectiveness has been empirically corroborated.

Furthermore, Table 1 provides a list of variables and their definition to improve readability.

3.1 Uncertainty measures

In order to carry the hypothesis test in (10), one has options for $U(\mathbf{x}_{\nu};\{\mathbf{p}_{l}\}_{l=1}^{L})$ other than the conditional variance. For DNNs designed for classification, mutual information has been recently proposed as a measure of uncertainty [38]

[TABLE]

where superscript $r$ indexes the pass of input $\mathbf{x}_{\nu}$ through the $r$th DNN realization with corresponding random output $\mathbf{y}_{\nu}^{(r)}:=[y_{\nu,1}^{(r)},y_{\nu,2}^{(r)},\ldots,y_{\nu,K}^{(r)}]^{\top}$ in a $K$-class classification task, and $H(.)$ is the entropy function222Entropy functions in (11) are also parameterized by $\{\mathbf{p}_{l}\}_{l=1}^{L}$, but we abbreviate them here as $H(\bar{\mathbf{y}}_{\nu})$ and $H(\mathbf{y}_{\nu}^{(r)})$.

[TABLE]

The test statistic in (10) requires finding $\{\mathbf{p}_{l}\}_{l=1}^{L}$ by solving

[TABLE]

which is highly non-convex. However, using Taylor’s expansion of the logarithmic terms in (12), one can approximate the mutual information in (11) with the variance score $\text{Tr}(\text{Cov}_{q_{\theta}(\omega)}[\mathbf{y}_{\nu}])$ in (10), where the conditioning on $\mathbf{x}_{\nu}$ has been dropped for brevity [38]. As a result, the optimization in (13) is approximated as

[TABLE]

To solve (14), one needs to express the objective in terms of the optimization variables ${\{\mathbf{p}_{l}\}}$ for all layers explicitly. To this end, the following section studies a two-layer network, whose result will then be generalized to deeper models.

3.2 *Simplification of the predictive variance *

Aiming at a convenient expression for the cost in (14), consider first a two-layer network with input-output (I/O) relationship333Derivations in this section carry over readily to a more general I/O \mathbf{y}_{\nu}=\sigma_{\text{softmax}}\Big{(}\mathbf{W}_{2}\sigma(\mathbf{W}_{1}\mathbf{x}_{\nu}+\mathbf{b}_{1})+\mathbf{b}_{2}\Big{)} with $\mathbf{b}_{1}$ and $\mathbf{b}_{2}$ deterministic.

[TABLE]

where $\mathbf{W}_{1},\mathbf{W}_{2}$ are random matrices corresponding to the weights of the two layers as in (6), while $\sigma_{\text{softmax}}$ is the softmax memoryless nonlinearity

[TABLE]

with $\mathbf{u}:=[u_{1},u_{2},\ldots,u_{K}]^{\top}$, and the inner $\sigma$ in (15) models a general differentiable nonlinearity such as tanh. Although differentiability of the nonlinearities is needed for the derivations in this section, the general idea will be later tested on networks with non-differentiable nonlinearities (such as ReLU) in the experiments.

Given trained weights $\{\mathbf{W}_{l}^{(TR)}\}_{l=1}^{2}$, and using (4) and (6), the random weight matrices are found as

[TABLE]

where $\mathbf{S}_{l}=\text{diag}(\mathbf{z}_{l})$ denotes the random sampling matrix with pseudo-inverse diagonal mean given by \mathbf{D}_{l}=\text{diag}^{\dagger}\Big{(}\mathbb{E}_{q_{z}(\mathbf{z}_{l};\mathbf{p}_{l})}[\mathbf{z}_{l}]\Big{)}. Since $\mathbb{E}[\mathbf{W}_{l}]\mathbf{x}_{(l-1)}=\mathbf{W}_{l}^{(TR)}\mathbf{x}_{(l-1)}$, the mean of $\mathbf{W}_{l}$ does not depend on $\mathbf{p}_{l}$, while its higher-order moments do.

Proposition 1. For the two-layer network in (15), the proposed minimization in (14) can be approximated by

[TABLE]

where $\gamma$ is a constant. The solution of (17) proceeds in two steps

[TABLE]

[TABLE]

where \gamma^{\prime}:=\gamma\text{Tr}(\text{ Cov}_{\mathbf{W}_{1}}[{\mathbf{W}_{1}\mathbf{x}}_{\nu}])\Big{|}_{\mathbf{p}_{1}=\mathbf{p}^{*}_{1}}.

Proof. See Appendix 8.1.

Remark. The cost in (17) approximates that in (14) by casting the overall uncertainty minimization as a weighted sum of layer-wise variances. In particular, $\mathbf{p}_{1}^{*}$ is the sampling probability vector that minimizes variance score of the first layer. It subsequently influences the regularization scalar $\gamma^{\prime}$ in minimizing the second layer variance, which yields the pmf vector $\mathbf{p}_{2}^{*}$. This can be inductively generalized to $L>2$ layers. As $L$ increases however, so do the number of cross terms. For simplicity and scalability, we will further approximate the per-layer minimization by dropping the regularization term, which leads to separable optimization across layers. This is an intuitively pleasing relaxation, because layer-wise variance is minimized under $\mathcal{H}_{0}$, which also minimizes the regularization weight $\gamma^{\prime}$.

The resulting non-regularized approximant of step 2 is

[TABLE]

generalizing to the $l$-th layer in an $L$-layer DNN as

[TABLE]

where $\mathbf{x}_{(l-1)}$ is the output of the $(l-1)$st layer, regardless of pmf vectors of other layers $\{\mathbf{p}_{l^{\prime}}\}_{l^{\prime}\neq l}$.

3.3 *Layer-wise variance minimization *

Here we will solve the layer-wise variance minimization in (18). Using (16), the cost can be upper bounded by

[TABLE]

where the last equality follows because the $C$ draws are iid with replacement, and the binary random variables $z_{l,i}$ reduce to Bernoulli ones with parameter $\pi_{l,i}=1-(1-p_{l,i})^{C}$; hence, for $\mathbf{x}_{(l-1),i}\neq 0$ it holds that $\mathbb{E}[{S}_{ii}^{2}{D}_{ii}^{2}]=1/\pi_{l,i}$ and $\mathbb{E}[{S}_{ii}{D}_{ii}]=1$, which implies that $\mathbb{E}[({S}_{ii}{D}_{ii}-1)^{2}]=(1/\pi_{l,i})-2+1$.

Using (19), the optimization in (18) can be approximately solved by a majorized surrogate as

[TABLE]

which is a convex problem that can be solved efficiently as elaborated next.

4 Solving layer-by-layer minimization

Consider rewriting the layer-wise variance minimization in (20) in a general form as

[TABLE]

where $\alpha_{i}:=x_{(l-1),i}^{2}$ for the $l$-th layer. Over the feasible set of the probability simplex, the cost in (21) has semi-definite Hessian; thus, it is convex, and can be solved by projected gradient descent iterations. However, $\mathbf{p}$ lies in the probability simplex space of dimension $h$, the number of hidden nodes in a given layer, and is typically very large. The large number of variables together with possible ill-conditioning can slow down the convergence rate.

To obtain a solver with quadratic convergence rate, we build on the fact that $h_{l}$ is usually very large, which implies that $p_{i}\ll 1$ for the practical setting at hand. Using the inequality $1-(1-p_{i})^{C}\geq 1-e^{-Cp_{i}}$, the cost in (21) can then be tightly upperbounded, which leads to majorizing (21) as

[TABLE]

The KKT conditions yield the optimal solution of the convex problem in (22), as summarized next.

Proposition 2. *The optimization in (22) can be solved with quadratic convergence rate, and the optimum is given by *

[TABLE]

where $\rho^{*}$ is the solution to the following root-finding problem

[TABLE]

Proof. See Appendix 8.2.

4.1 Approximate variance minimization for small $C$

For small values of $C$, it holds that $(1-p_{i})^{C}>1-Cp_{i}$; hence, the Bernoulli parameter $\pi_{i}=1-(1-p_{i})^{C}$ can be approximated by its upperbound $Cp_{i}>\pi_{i}$. With this we can approximate the cost in (20), as

[TABLE]

Using the Lagrangian and the KKT conditions, we then find $p_{i}^{*}={\sqrt{\alpha_{i}}}/{\sum_{j}{\sqrt{\alpha_{j}}}}$, which for the $l$-th layer is expressible as

[TABLE]

This approximation provides analytical justification for the heuristic approach in [44], where it is proposed to sample with probabilities proportional to the magnitude of the hidden unit outputs. However, there remains a subtle difference, which will be clarified in Section 6.

Approximating (22) with (24) can be loose for large values of $C$, which motivates our next approximation.

4.2 Approximate variance minimization for large $C$

Building on the tight approximation in (22), one can further approximate the variance for large $C$ as

[TABLE]

where we have used $(1-\delta)^{-1}\simeq 1+\delta$ as a tight approximation for $0<\delta\ll 1$. This leads to the minimization

[TABLE]

which again is a convex problem, whose solution can be obtained using the KKT conditions that lead to

[TABLE]

where $\lambda$ is the Lagrange multiplier. Under the simplex constraint on the $\{p_{i}\}$, this leads to the optimal

[TABLE]

with $[\,.\,]_{+}$ denoting the projection on the positive orthant, and the normalization constant $\beta:=-\ln\lambda/C$ having optimal value

[TABLE]

Although the solution to the fixed point condition cannot be obtained at one shot, and may require a few iterations to converge, in practice we only perform it once and settle with the obtained approximate solution$\{\hat{p}^{*}_{i}\}_{i=1}^{h}$.

5 Practical issues

The present section deals with efficient implementation of the proposed approach in practice, and establishes links with state-of-the-art randomization-based detection methods.

5.1 Efficient implementation via non-uniform dropout

The proposed defense builds on modeling the variational pdf $q_{\theta}(\omega)$ using a sampling-with-replacement process. Performing the proposed process however, may incur overhead complexity during inference when compared to the inexpensive dropout alternative outlined in Sec. 2.1. To reduce this complexity, one can implement our approach using efficient approximations, while leveraging the sampling probabilities learned through our uncertainty minimization.

Reflecting on the binary variables $\{z_{l,i}\}$ that model the pickup of the hidden node $i$ in the overall sampling process in (9), one can approximate the joint pmf of $\{z_{l,i}\}_{i=1}^{h_{l}}$ as

[TABLE]

where random variables $\{z_{l,i}\}_{i}$ are now viewed as approximately independent non-identical Bernoulli variables with parameters $\{\pi_{l,i}\}_{i=1}^{h_{l}}$; that is, $z_{l,i}\sim$ Bernoulli$(\pi_{l,i})$ for $i=1,\ldots,h_{l}$, where $\pi_{l,i}=1-(1-p_{l,i})^{C}$.

Although (27) is an approximation, it provides insight but also an efficient implementation of the sampling process. In fact, the proposed optimization in (21) can now be viewed as an optimization over the non-uniform dropout probabilities, coupled implicitly through the hyper-parameter $C$, whose selection guarantees a certain level of randomness. This is to be contrasted with finding optimal dropout probabilities - a task requiring grid search over an $h_{l}$-dimensional space for layer $l$, where $h_{l}$ can be hundreds of thousands to millions in CNNs classifying high-quality images. Interestingly, the proposed convex optimization simplifies the high-dimensional grid-search into a scalar root-finding task, whose solution can be efficiently found with super-linear (quadratic) convergence rate.

5.2 Placement and adjustment of the sampling units

It has been argued that CNN layers at different depths can provide extracted features with variable levels of expressiveness [48]. On a par with this, one can envision the defense potential at different depths by incorporating sampling units across say $B$ blocks of the network as listed in Tables 6 and LABEL:tab:resnet34. In particular, the dropout defense has been mostly utilized at the last layer after flattening [38], whereas here we consider the potential of sampling at earlier layers that has gone mostly under-explored so far. This can in turn result in DNN-based classifiers with robustness to adversarial attacks, as optimal sampling at the initial layers maybe crucial for correct detection of the adversarial input. We henceforth refer to a DNN (or CNN) equipped with random sampling as the detection network, and the original one without the sampling units as the full network.

Similar to the pick up probability $\pi$ in uniform dropouts, the number of draws $C$ in our approach is a hyper parameter that controls the level of randomness present in the detection network. Qualitatively speaking, the smaller number of units (smaller $C$) is picked per layer, the larger ‘amount of randomness’ emerges (further $\pi_{l,i}$ is from $1$). This can lead to forward propagating not as informative (under-sampled) features, meaning not representative of the clean image, and can thus cause unreliable detection. A large $C$ on the other hand, increases the probability to pick up units per layer, which requires a large number of MC realizations for reliable detection, otherwise small randomness will lead to miss-detection. At the extreme, very large $C$ renders the detection and full networks identical, thus leading to unsuccessful detection of adversarial inputs. In a nutshell, there is a trade-off in selecting $C$, potentially different for the initial, middle, and final layers of a given CNN.

Fig. 1 categorizes existing and the novel randomization-based approaches to detecting adversarial inputs.

Uniform dropout. In this method, units are independently dropped w.p. $1-\pi$, and sampled (picked) w.p. $\pi$ $\forall l,i$.

Non-uniform dropout using variance minimization. Dropout here follows the scheme in subsection 5.1, for which we pursue the following two general cases with deterministic and dynamic probabilities.

(C1) Variance minimization with fixed probabilities. In this case, the image is first passed through the full network to obtain the values $\{x_{(l-1),i}\}$ of the unit outputs per hidden layer. These are needed to determine the non-uniform dropout probabilities $1-p_{l,i}$ (thus $\pi_{l,i}$ and then the index of the units to sample) via exact, linear, or logarithmic approximations given in (23), (25) and (26), respectively, refered to as VM-exact, VM-lin, and VM-log; see Fig. 2-a.

Despite parallel MC passes in the proposed class of sampling with fixed probabilities (step 3 in Fig. 2-a), the first step still imposes a serial overhead in detection since the wanted probabilities must be obtained using a pass through the full network. Our approach to circumventing this overhead is through approximation using the following class of sampling with dynamic probabilities.

(C2) Variance minimization with dynamic probabilities. Rather than finding the sampling probabilities beforehand, $p_{l,i}^{(r)}$ are determined on-the-fly as the image is passed through the detection network with the units sampled per layer. As a result, the observed unit values are random (after passing through at least one unit sampled), and are different across realizations. In order to mitigate solving many optimization problems, variance minimization with dynamic probabilities is only implemented via linear and logarithmic approximations (25) and (26); and are referred to as DVM-lin and DVM-log, respectively; see Fig. 2-b.

It is interesting to note that DVM-lin corresponds to the proposed stochastic activation pruning (SAP) in [44], with

[TABLE]

where $x_{(l-1),i}^{(r)}$ is the output of the $i$-th activation unit of the $l$-th layer in the $r$-th realization for input $\mathbf{x}$.

Figure 1 provides an overview of the sampling methods, while Algorithms 1 and 2 outline the two proposed variance minimization-based detection methods in pseudocode.

6 Numerical tests

In this section, we test the effectiveness of the proposed sampling method for detecting various adversarial attacks on CNNs used for image classification. In order to address the raised issue in [13], classification of the CIFAR10 image dataset using ResNet20 as well as the high-resolution cats-and-dogs images using ResNet34 networks [49] are tested. A short summary of the two networks and datasets can be found in Tables II, 6 and LABEL:tab:resnet34. In order to investigate the issue around placement of the sampling units, we will place them after ReLU activation layers in different “blocks” ($B$) of the ResNet20 and ResNet34 networks, as listed in Tables 6 and LABEL:tab:resnet34. Numerical tests are made available online.444https://github.com/FatemehSheikholeslami/variance-minimization