Defending Against Universal Perturbations With Shared Adversarial Training

TL;DR

This paper enhances adversarial training methods to better defend deep neural networks against universal perturbations, which are more challenging and can be made perceptible with improved robustness.

Contribution

It demonstrates that adversarial training is more effective against universal perturbations and proposes an extension to balance robustness and performance.

Findings

Universal perturbations become perceptible after robust training.

Adversarial training reduces success rate of universal attacks.

Trade-off between robustness and accuracy is addressed.

Abstract

Classifiers such as deep neural networks have been shown to be vulnerable against adversarial perturbations on problems with high-dimensional input space. While adversarial training improves the robustness of image classifiers against such adversarial perturbations, it leaves them sensitive to perturbations on a non-negligible fraction of the inputs. In this work, we show that adversarial training is more effective in preventing universal perturbations, where the same perturbation needs to fool a classifier on many inputs. Moreover, we investigate the trade-off between robustness against universal perturbations and performance on unperturbed data and propose an extension of adversarial training that handles this trade-off more gracefully. We present results for image classification and semantic segmentation to showcase that universal perturbations that fool a model hardened with adversarial training become clearly perceptible and show patterns of the target scene.

Tables

Table 1. Table A1: Universal robustness and classification accuracy of ResNet20 trained with standard adversarial training ( s = 1 𝑠 1 s=1 ) and shared adversarial training s ∈ { 8 , 64 } 𝑠 8 64 s\in\{8,64\} against S-PGD universal perturbations on CIFAR10 under different range of attack parameters ε 𝜀 \varepsilon and σ 𝜎 \sigma . The pictorial representation of these entries are depicted in Figure 3 (left). The bold entries represent the model trained with shared adversarial training s = 64 𝑠 64 s=64 that yields threefold increase in robustness when compared to undefended model with a drop of less than 3.5 % percent 3.5 3.5\% in accuracy.

$\sigma$	$s$		$\epsilon =2$	$\epsilon =4$	$\epsilon =6$	$\epsilon =8$	$\epsilon =10$	$\epsilon =14$	$\epsilon =18$	$\epsilon =22$	$\epsilon =26$
0.3	1	Acc.(%)	92.16	90.42	88.76	84.02	78.54	73.61	70.32	67.63	61.73
		${\epsilon }_{uni}\left(\delta =0.75\right)$	28.14	33.62	34.37	40.59	38.85	41.59	56.03	48.06	48.06
	8	Acc.(%)	93.04	92.05	91.34	90.75	89.09	85.87	82.68	79.69	76.35
		${\epsilon }_{uni}\left(\delta =0.75\right)$	23.41	26.89	32.62	34.12	32.87	40.59	45.82	47.31	54.54
	64	Acc.(%)	93.72	93.36	92.84	92.47	91.90	90.89	88.93	86.13	83.89
		${\epsilon }_{uni}\left(\delta =0.75\right)$	19.42	22.66	25.90	27.89	32.37	39.10	40.34	42.58	46.32
0.5	1	Acc.(%)	91.77	89.70	87.60	85.44	82.64	71.81	66.61	64.46	62.16
		${\epsilon }_{uni}\left(\delta =0.75\right)$	30.63	38.10	39.35	42.33	47.31	61.26	55.78	93.27	63.75
	8	Acc.(%)	92.62	91.69	90.86	90.11	89.04	86.16	82.92	80.54	75.99
		${\epsilon }_{uni}\left(\delta =0.75\right)$	23.16	29.88	33.87	36.61	38.10	42.83	43.83	53.54	64.00
	64	Acc.(%)	93.55	93.09	92.50	91.99	91.62	90.58	88.52	86.65	84.13
		${\epsilon }_{uni}\left(\delta =0.75\right)$	20.67	23.91	28.14	29.88	38.10	37.60	41.34	47.31	51.05
0.7	1	Acc.(%)	91.55	89.07	86.58	84.47	81.97	78.75	73.29	64.15	63.63
		${\epsilon }_{uni}\left(\delta =0.75\right)$	31.63	42.08	39.84	45.82	46.82	54.04	63.75	95.63	89.65
	8	Acc.(%)	92.51	91.29	90.23	89.12	88.08	85.70	83.74	80.27	78.00
		${\epsilon }_{uni}\left(\delta =0.75\right)$	26.40	30.38	35.11	39.84	44.08	45.07	45.82	50.55	54.54
	64	Acc.(%)	93.34	92.81	92.27	91.67	91.30	89.94	88.25	85.96	83.94
		${\epsilon }_{uni}\left(\delta =0.75\right)$	20.67	23.91	27.89	30.88	36.11	44.08	43.58	51.05	50.80
0.9	1	Acc.(%)	91.45	88.54	85.51	82.97	80.38	76.35	72.94	70.14	68.20
		${\epsilon }_{uni}\left(\delta =0.75\right)$	32.87	38.85	46.07	50.55	54.54	58.27	63.25	64.50	59.77
	8	Acc.(%)	92.27	90.97	89.89	88.63	86.89	84.25	81.63	79.13	76.56
		${\epsilon }_{uni}\left(\delta =0.75\right)$	26.40	33.12	37.10	40.34	44.08	45.82	47.81	55.28	58.02
	64	Acc.(%)	93.18	92.61	92.12	91.42	90.85	89.32	87.41	85.26	83.10
		${\epsilon }_{uni}\left(\delta =0.75\right)$	22.16	25.40	29.63	32.87	36.61	43.33	46.07	45.32	54.04

Table 2. Table A2: Universal robustness and classification accuracy of WRN-50-2-bottleneck trained with standard adversarial training ( s = 1 𝑠 1 s=1 ) and shared adversarial training ( s = 32 𝑠 32 s=32 ) against S-PGD universal perturbations on a subset of ImageNet (200 classes) under different range of attack parameters ε 𝜀 \varepsilon and σ 𝜎 \sigma . The pictorial representation of these entries are depicted in Figure 4 . The bold entries represent the model trained with shared adversarial training s = 32 𝑠 32 s=32 that yields threefold increase in robustness when compared to undefended model with a drop of less than 5 % percent 5 5\% in accuracy.

$\sigma$	$s$		$\epsilon =2$	$\epsilon =4$	$\epsilon =6$	$\epsilon =8$	$\epsilon =10$	$\epsilon =14$	$\epsilon =18$	$\epsilon =22$	$\epsilon =26$
0.5	1	Acc.(%)	75.81	74.49	71.14	71.48	68.78	66.25	64.05	59.04	57.54
		${\epsilon }_{uni}\left(\delta =0.75\right)$	17.92	18.92	23.90	17.92	13.94	20.66	20.41	29.38	25.89
	32	Acc.(%)	77.50	77.23	75.46	72.77	68.43	64.21	56.48	54.58	48.30
		${\epsilon }_{uni}\left(\delta =0.75\right)$	13.44	15.93	17.43	12.70	18.67	28.13	33.36	43.33	45.57
1.0	1	Acc.(%)	74.83	71.84	67.79	66.72	63.51	60.38	58.49	57.23	55.19
		${\epsilon }_{uni}\left(\delta =0.75\right)$	20.17	23.90	24.90	28.88	27.89	30.13	30.62	32.12	32.37
	32	Acc.(%)	77.41	76.41	75.22	73.86	72.74	70.04	66.97	62.36	56.99
		${\epsilon }_{uni}\left(\delta =0.75\right)$	14.94	17.43	19.67	21.41	25.64	28.88	30.13	29.38	33.36

Equations

$${\rho }_{adv}={\rho }_{heap}^{(1)}\ge {\rho }_{heap}^{(2)}\ge {\rho }_{heap}^{(4)}\ge \cdots \ge {\rho }_{heap}^{(d)}\ge {\rho }_{uni}(\sigma ,\mathcal{S})$$

$${\tilde{\rho }}_{adv}={\tilde{\rho }}_{heap}^{(1)}\ge {\tilde{\rho }}_{heap}^{(2)}\ge {\tilde{\rho }}_{heap}^{(4)}\ge \cdots \ge {\tilde{\rho }}_{heap}^{(d)}\ge {\tilde{\rho }}_{uni}(\sigma ,\mathcal{S})$$

Full text

1 Introduction

While deep learning is relatively robust to random noise [11], it can be easily fooled by adversarial perturbations [44]. These perturbations are generated by adversarial attacks [15, 31, 5] that generate perturbed versions of the input which are misclassified by a classifier and remain quasi-imperceptible for humans. There have been different approaches for explaining properties of adversarial examples and provide rationale for their existence in the first place [15, 45, 12, 13]. Moreover, these perturbations have been shown to be relatively robust against various kinds of image transformations and are even successful when placed as artifacts in the physical world [21, 43, 10, 4]. Thus, adversarial perturbations might pose a safety and security risk for autonomous systems and also reduce trust on the models that are in principle vulnerable to these perturbations.

Several methods have been proposed for increasing the robustness of deep networks against adversarial examples, such as adversarial training [15, 22], virtual adversarial training [28], ensemble adversarial training [46], defensive distillation [36, 35], stability training [50], robust optimization [25], Parseval networks [7] and alternatively detecting and rejecting them as malicious [26]. While some of these approaches improve robustness against adversarial examples to some extent, the classifier remains vulnerable against adversarial perturbations on a non-negligible fraction of the inputs for all defenses [3, 47].

Most work has focused on increasing robustness in image classification tasks, where the adversary can choose a data-dependent perturbation for each input. This setting is very much in favor of the adversary since the adversary can craft a high-dimensional perturbation “just” to fool a model on a single input. In this work, we argue that limited success in increasing the robustness under these conditions does not necessarily imply that robustness can not be achieved in other settings. Specifically, we focus on robustness against input-agnostic perturbations, namely universal perturbations [29], where the same perturbation needs to fool a classifier on many inputs. Moreover, we investigate robustness against such perturbations in dense prediction tasks such as semantic image segmentation, where a perturbation needs to fool a model on many decisions, e.g., the pixel-wise classifications. Data-dependent adversarial attacks need to know their input in advance and require online computation to generate perturbations for every incoming input whereas universal attacks work on unseen inputs.

Prior work has shown that standard models are vulnerable to both universal perturbations, which mislead a classifier on the majority of the inputs [29, 32], and to adversarial perturbations on semantic segmentation tasks [14, 48, 6].

The study of robustness against universal perturbations is important since they pose a realistic threat-model for certain physical-world attacks: for instance, Li et al. [23] show that an adversary could mount a semi-transparent adversarial sticker on a physical camera which effectively adds a universal perturbation to each unseen camera image. It was demonstrated by Metzen et al. [27] that such universal perturbations can hide nearby pedestrians in semantic segmentation which may allow deceiving an emergency braking system and would also pose a threat in surveillance scenarios. However, these and similar results have been achieved for undefended models. In this work, we focus on the case where models have been “hardened” by a defense mechanism, particularly adversarial training. While this technique can considerably increase robustness, there is an implicit trade-off between robustness against perturbations and high performance on unperturbed inputs. We show that explicitly tailoring adversarial training for universal perturbations allows handling this trade-off more gracefully.

Our main contributions are as follows: (1) We propose shared adversarial training, an extension of adversarial training that handles the inherent trade-off between accuracy on clean examples and robustness against universal perturbations more gracefully. (2) We evaluate our method on CIFAR10, a subset of ImageNet (with 200 classes), and Cityscapes to demonstrate that universal perturbations for the defended models become clearly perceptible as shown in Figure 1. (3) We are the first to scale defenses based on adversarial training to semantic segmentation. (4) We demonstrate empirically on CIFAR10 that the proposed technique outperforms other defense mechanisms [30, 37] in terms of robustness against universal perturbations.

2 Related Work

In this section, we review related work on the study of universal perturbations and adversarial perturbations for semantic image segmentation.

2.1 Universal Perturbations

Different methods for generating universal perturbations exist: Moosavi-Dezfooli et al. [29] uses an extension of the DeepFool adversary [31] to generate perturbations that fool a classifier on a maximum number of inputs from a training set. Metzen et al. [27] proposed a similar extension of the basic iterative adversary [22] for generating universal perturbations for semantic image segmentation. In contrast to former works, Mopuri et al. [33] proposed Fast Feature Fool, a data-independent approach for generating universal perturbations. In follow-up work [32], they show similar fooling rates of data-independent approaches as have been achieved by Moosavi-Dezfooli et al. [29]. Khrulkov and Oseledets [19] show a connection between universal perturbations and singular vectors. In another line of work, Hayes and Danezis [16], Mopuri et al. [40], and Poursaeed et al. [38] proposed generative models that can be trained to generate a diverse set of (universal) perturbations.

An analysis of universal perturbation and their properties is provided by Moosavi-Dezfooli et al. [30]. They connect the robustness to universal perturbations with the geometry of the decision boundary and prove the existence of small universal perturbation provided the decision boundary is systematically positively curved. Jetley et al. [18] build upon this work and provide evidence that directions in which a classifier is vulnerable to universal perturbations coincide with directions important for correct prediction on unperturbed data. They follow that predictive power and adversarial vulnerability are closely intertwined.

Prior procedures on robustness against universal perturbations define a distribution over (approximately optimal) such perturbations for a model (either by precomputing and random sampling [29], by learning a generative model [16], or by collecting an increasing set of universal perturbations for model checkpoints during training [37]), fine-tune model parameters to become robust against this distribution of perturbations, and (optionally) iterate. These procedures increase robustness against universal perturbations slightly, however, not to a satisfying level. This is probably caused by the model overfitting to the fixed distribution of universal perturbations that do not change during the optimization process. However, re-computing universal perturbations in every mini-batch anew is prohibitively expensive. In this work, we propose a method that can be performed efficiently by computing shared perturbations on each mini-batch and using them in adversarial training, i.e., the shared perturbations are computed on-the-fly rather than precomputed as in prior work [29, 37]. Concurrent to our work, Shafahi et al.[42] recently proposed “universal adversarial training” where updates of the neural network’s parameters and the universal perturbation happen concurrently. This reduces the overhead of determining a universal perturbation anew for every mini-batch; however, it is unclear if such an incrementally updated universal perturbation can track the changes of the network’s weights sufficiently.

Alternative defense approaches add additional components to the model: Ruan and Dai [41] proposed to identify and reject universal perturbations by adding shadow classifiers, while Akhtar et al. [1] proposed to prepend a subnetwork in front of the model that is used to compensate for the added universal perturbation by detecting and rectifying the perturbation. Both methods have the disadvantage that the model becomes large and thus inference more costly. More severely, it is assumed that the adversary is not aware of the defense mechanism and it is unclear if a more powerful adversary could not fool the defense mechanism.

2.2 Adversarial Perturbations for Semantic Image Segmentation

Methods for generating adversarial perturbations have been extended to structured and dense prediction tasks like semantic segmentation and object detection [14, 48, 6]. Metzen et al. [27] even showed the existence of universal perturbations which result in an arbitrary target segmentation of the scene which has nothing in common with the scene a human perceives. A comparison of the robustness of different network architectures has been conducted by Arnab et al. [2]: they found that residual connections and multiscale processing actually increase robustness of an architecture, while mean-field inference for Dense Conditional Random Fields only masks gradient but does not increase robustness itself. In contrast to their work, we focus on modifying the training procedure for increasing robustness. Both approaches could be combined in the future.

3 Preliminaries

In this section, we introduce basic terms and notations relevant for this work. We aim to defend against an adversary under white-box attack settings. Please refer to Section A.1 in the supplementary material for details on capabilities of the adversary and the threat model.

3.1 Risks

Let $L$ be a loss function (categorical crossentropy throughout this work), $\mathcal{D}$ be a data distribution, and $\theta$ be the parameters of a parametric model $f_{\theta}$. Here, we define the risk $\rho(\theta)$ as the expected loss of the model $f_{\theta}$ for a data distribution. The following risks are relevant for this work (we extend the definitions of Uesato et al. [47]):

Expected Risk: $\rho_{exp}(\theta)=\mathbb{E}_{(x,y)\sim\mathcal{D}}\,L(\theta,x,y)$ 2. 2.

Adversarial Risk:

$\rho_{adv}(\theta,\mathcal{S})=\mathbb{E}_{(x,y)\sim\mathcal{D}}\left[\sup\limits_{\xi(x)\in\mathcal{S}}L(\theta,x+\xi(x),y)\right]$ 3. 3.

Universal Adversarial Risk:

$\rho_{uni}(\theta,\mathcal{S})=\sup\limits_{\xi\in\mathcal{S}}\mathbb{E}_{(x,y)\sim\mathcal{D}}\left[L(\theta,x+\xi,y)\right]$

Here, $\xi(x)$ denotes an adversarial perturbation, $\xi$ a universal perturbation, and $x+\xi(x)$ an adversarial example. The set $\mathcal{S}$ defines the space from which perturbations may be chosen. We would like to note that adversarial and universal risk are not equivalent since in the former case, $\xi(x)$ depends on the specific $x$ sampled from $\mathcal{D}$, while the latter, $\xi$ needs to generalize over the entire data distribution $\mathcal{D}$.

3.2 Adversaries

Since the worst-case perturbation $\xi(x)$ cannot be computed efficiently in typical settings, one needs to resort to an adversary which aims at finding a strong perturbation $\xi(x)$. Note that this corresponds to searching for a tight lower bound of $\rho_{adv}$. We define an adversary as a function $f_{adv}:\mathcal{D}\times\Theta\mapsto\mathcal{S}$, which maps a data point and model parameters $\theta$ onto a perturbation $\xi(x)$ that maximizes a loss $L_{adv}(\theta,x+\xi(x),y)$111We note that one may choose $L_{adv}=L$ or one may also choose, e.g., $L$ to be the 0-1 loss and $L_{adv}$ be a differentiable surrogate loss.. While different options for the adversaries $f_{adv}$ exist [15, 31, 5, 29, 32], we focus on projected gradient descent (PGD) [25, 21], as it provides in our experience a good trade-off between being computationally efficient and powerful. PGD initializes $\xi^{(0)}$ uniformly at random in $\mathcal{S}$ (or subset of $\mathcal{S}$) and performs $K$ iterations of the following update:

$\xi^{(k+1)}=\Pi_{\mathcal{S}}\left[\xi^{(k)}+\alpha_{k}\cdot\text{sgn}(\nabla_{x}L_{adv}(\theta,x+\xi^{(k)},y)\right],$ where $\Pi_{\mathcal{S}}$ denotes a projection on the space $\mathcal{S}$ and $\alpha_{k}$ denotes a step-size. Similarly, a targeted attack where the model shall output the target class $y_{t}$ can be obtained by setting $\alpha_{k}$ to $-\alpha_{k}$ and $y$ to $y_{t}$.

Similar to a standard adversary, we define a universal adversary denoted by $f_{uni}$ as function mapping model parameters $\theta$ onto perturbation $\xi$ with the objective of maximizing $\mathbb{E}_{(x,y)\sim\mathcal{D}}\left[L_{adv}(\theta,x+\xi,y)\right]$. One can modify PGD into a universal adversary by using the loss $L_{uni}(\theta,\{x_{i},y_{i}\}_{i=1}^{m},\xi)=\frac{1}{m}\sum_{i=1}^{m}L_{adv}(\theta,x_{i}+\xi,y_{i})$. If the number of data points $m$ is large (which is typically required for finding universal perturbations that generalize well to unseen data), one can employ stochastic PGD, where in every iteration $k$, a set of $\tilde{m}_{k}$ data points is sampled and $L_{uni}$ is only evaluated on this subset of data points.

4 Shared Adversarial Training

We connect the above risks to show that adversarial training optimizes a loose upper bound on the universal risk and motivate shared adversarial training, an extension of adversarial training that aims at maximizing robustness against universal perturbations. We show that this method minimizes an upper bound on the universal risk which is tighter than the one used in adversarial training.

4.1 Relationship between Risks

We show the following inequalities for the risks:$\rho_{exp}(\theta)\leq\rho_{uni}(\theta,\mathcal{S})\leq\rho_{adv}(\theta,\mathcal{S})\;\forall\theta\;\forall\mathcal{S}\supset\{\mathbf{0}\}$. To see the validity of these inequalities, we set $\mathcal{S}=\{\mathbf{0}\}$ to obtain $\rho_{uni}(\theta,\mathcal{S})=\rho_{exp}(\theta)$ (and $\mathcal{S}\supset\{\mathbf{0}\}$ can only increase $\rho_{uni}(\theta,\mathcal{S})$). For the second inequality, assume $\rho_{adv}<\rho_{uni}$. Let $\xi$ be one of the multiple universal perturbations that maximize $\rho_{uni}$. Since $\xi$ is an element of $\mathcal{S}$, we could certainly set $\xi(x)=\xi$ $\forall x$ in the definition of the adversarial risk. This would result in $\rho_{adv}=\rho_{uni}$. This completes the proof by contradiction, and thus $\rho_{adv}$ can only be larger than or equal to $\rho_{uni}$ in general.

The objective of adversarial training is defined as minimizing the loss function $\sigma\cdot\rho_{adv}(\theta,\mathcal{S})+(1-\sigma)\cdot\rho_{exp}(\theta)$, where $\sigma$ controls the trade-off between robustness and performance on unperturbed inputs. We note that if one is interested in minimizing the universal adversarial risk $\rho_{uni}$, then using $\rho_{adv}$ in adversarial training with $\sigma=1$ corresponds to minimizing an upper bound of $\rho_{uni}$ because $\rho_{uni}(\theta,\mathcal{S})\leq\rho_{adv}(\theta,\mathcal{S})$, provided that the adversaries find perturbations that are sufficiently close to the optimal perturbations. On the other hand, standard empirical risk minimization ERM ($\sigma=0$), which minimizes the empirical estimate of $\rho_{exp}$, corresponds to minimizing a lower bound. As shown in previous work [15, 31, 5], this does confer only little robustness against (universal) perturbations. For $0<\sigma<1$, adversarial training corresponds to minimizing a convex combination of the upper bound $\rho_{adv}$ and the lower bound $\rho_{exp}$ but does not directly optimize on $\rho_{uni}$. As we show in Section 6, this standard version of adversarial training already provides strong robustness against universal perturbations at the cost of reducing performance on unperturbed data considerably.

4.2 Method

Directly employing $\rho_{uni}$ in adversarial training is infeasible since evaluating $\rho_{uni}(\theta,\mathcal{S})$ with an adversary $f_{uni}$ in every mini-batch is prohibitively expensive (because it requires large $m$). Hence, it would be desirable to use an upper bound of $\rho_{uni}$ in adversarial training that is tighter than $\rho_{adv}$ but cheaper to approximate than $\rho_{uni}$.

For this, we propose to use a so-called heap adversary, which we define as a function $f_{heap}:\mathcal{D}^{m}\times\Theta\mapsto\mathcal{S}$ that maps a set of $m$ data points and model parameters $\theta$ onto a perturbation $\xi$. We use $L_{uni}(\theta,\{x_{i},y_{i}\}_{i=1}^{m},\xi)=\frac{1}{m}\sum_{i=1}^{m}L_{adv}(\theta,x_{i}+\xi,y_{i})$ as loss function for the heap adversary. However, in contrast to a universal adversary, we do not require a heap adversary to find perturbations that generalize to unseen data. This allows choosing $m$ relatively small.

More specifically, we split a mini-batch consisting of $d$ data points into $d/s$ heaps (subsets of the mini-batch) of size $s$ (we denote $s$ as sharedness). Rather than using the adversary $f_{adv}$ for computing a perturbation on each of the $d$ data points separately, we employ a heap adversary $f_{heap}$ for computing $d/s$ shared perturbations on the heaps with $m=s$. Thereupon, these perturbations are broadcasted to all $d$ data points by repeating each of the shared perturbations $s$ times for all elements of the heap. Employing this heap adversary implies a risk $\rho_{heap}^{(s)}$. We propose to use $\rho_{heap}^{(s)}$ in adversarial training when aiming at defending against universal perturbations and denote the resulting procedure as shared adversarial training. This entire process is illustrated in Figure 2. We can obtain the following relationship for $s=2^{i}$ (please refer to Section A.2 for more details):

[TABLE]

Note that while all $\rho_{heap}^{(s)}$ are upper bounds on the universal risk $\rho_{uni}$, this does not imply that shared perturbations are strong universal perturbations. In contrast, the smaller $s$, the more “overfit” are the shared perturbations to the respective heap. However, $\rho_{heap}^{(s)}$ with $s\gg 1$ is typically a much tighter upper bound on $\rho_{uni}$ than $\rho_{adv}$ and can be approximated as efficiently as $\rho_{adv}$: for this, PGD is converted into a heap adversary by replacing $L_{adv}$ with $L_{uni}$. By appropriately reshaping and broadcasting perturbations, we can compute $d/s$ shared perturbations on the respective heaps of the mini-batch jointly by PGD with essentially the same cost as computing $d$ adversarial perturbations with PGD.

4.3 Adversarial Loss Function

We recall that $L_{uni}(\theta,\{x_{i},y_{i}\}_{i=1}^{m},\xi)=\frac{1}{m}\sum_{i=1}^{m}L_{adv}(\theta,x_{i}+\xi,y_{i})$. Because of limited capacity of the perturbation ($\xi\in\mathcal{S}$), there is “competition” between $m$ data points: the maximizers of $L_{adv}(\theta,x_{i}+\xi,y_{i})$ will typically be different and the data points will “pull” $\xi$ into different directions. Hence, using the categorical cross-entropy as a proxy for the 0-1 loss is problematic for untargeted adversaries: since we are maximizing the loss and the categorical cross-entropy has no upper bound, there is a winner-takes-all tendency where the perturbation is chosen such that it leads to highly confident misclassifications on some data points and to correct classification on other data points (this incurs higher cost than misclassifying more data points but with lower confidence).

To prevent this, we employ loss thresholding on the categorical cross-entropy $L$ to enforce an upper bound on $L_{adv}$: $L_{adv}(\theta,x,y)=\min(L(\theta,x,y),\kappa)$. We used $\kappa=-\log 0.2$, which corresponds to not encouraging the adversary to reduce confidence of the correct class below $0.2$. A similar loss thresholding was also proposed by Shafahi et al. [42] concurrently. Besides, we also incorporate label smoothing and use the soft targets for the computation of loss in all our experiments.

5 Robustness Evaluation

In this section, we define the measure of robustness used in the experiments and detail how we approximate it.

5.1 Definition of Robustness

For the special case of the 0-1 loss, an $n$-dimensional input $x$, and $\mathcal{S}=\mathcal{S}(\varepsilon)=[-\varepsilon,\varepsilon]^{n}$, we define the adversarial robustness as the smallest perturbation magnitude $\varepsilon$ that results in an adversarial risk (misclassification rate) of at least $\delta$. More formally:

$\varepsilon_{adv}(\delta)=\operatorname*{arg\,min}\limits_{\varepsilon}\rho_{adv}(\theta,\mathcal{S}(\epsilon))\text{ s.t. }\rho_{adv}(\theta,\mathcal{S}(\epsilon))>\delta.$

In other words, there are perturbations $\xi(x)$ with $||\xi(x)||_{\infty}<\varepsilon_{adv}(\delta)$ that result in a misclassification rate of at least $\delta$. Analogously, we can also define the universal robustness as

$\varepsilon_{uni}(\delta)=\operatorname*{arg\,min}\limits_{\varepsilon}\rho_{uni}(\theta,\mathcal{S}(\epsilon))\text{ s.t. }\rho_{uni}(\theta,\mathcal{S}(\epsilon))>\delta.$

Here, a perturbation $\xi$ with $||\xi||_{\infty}<\varepsilon_{uni}(\delta)$ exists that results in a misclassification rate of at least $\delta$.

5.2 Quantifying Robustness

Since the exact evaluation of $\varepsilon_{uni}(\delta)$ is intractable for our settings, we use an upper bound on the actual robustness $\varepsilon_{uni}(\delta)$ instead. For this, we tuned the PGD adversary as follows to make it more powerful (and thus the upper bound more tight): we performed a binary search of $b$ iterations for perturbation magnitude $\varepsilon$ of $\mathcal{S}(\varepsilon)$, i.e., the bound in the $l_{\infty}$ norm on the perturbation, on the interval $\varepsilon\in[0,255]$. In every iteration, we used the step-size annealing schedule $\alpha_{k}=\frac{\beta\varepsilon\gamma^{k}}{\sum_{j=0}^{K-1}\gamma^{j}}$ which guarantees that $\sum_{j=0}^{K-1}\alpha_{k}=\beta\varepsilon$. If a perturbation with misclassification rate $\delta$ is found in an iteration, the next iteration of binary search continues on the lower half of the interval for $\varepsilon$, otherwise on the upper half. The reported robustness is the smallest perturbation found in entire procedure that achieves a misclassification rate of $\delta$. Note that this procedure was only used for evaluation; for training we used a predefined $\varepsilon$ and constant step-size $\alpha_{k}$.

6 Experimental Results

We present experimental results of shared adversarial training on robustness against universal perturbations in both image classification and semantic segmentation tasks. We extended the PGD implementation of Cleverhans [34] such that it supports shared adversarial perturbations and loss clipping as discussed in Section 4. For quantifying robustness, we extended Foolbox [39] such that universal perturbations (with minimal $l_{\infty}$ norm) that achieve a misclassification rate of at least $\delta$ can be searched.

6.1 Experiments on CIFAR10

We present results on CIFAR10 [20] for ResNet20 [17] with 64-128-256 feature maps per stage. For evaluating robustness, we generate $f_{uni}$ using stochastic PGD on 5000 validation samples with mini-batches of size $\tilde{m}_{k}=16$ and evaluated on $512$ test samples. We used $b=10$ binary search iterations, $K=200$ S-PGD iterations, and the step-size schedule values $\gamma=0.975$ and $\beta=4$. We pretrained ResNet20 with standard regularized empirical risk minimization (ERM) and obtained an accuracy of $93.25\%$ on clean data and a robustness against universal perturbations of $\varepsilon_{uni}(\delta=0.75)=14.9$.

In general, we are interested in models that increase the robustness without decreasing the accuracy on clean data considerably. We consider this as a multi-objective problems with two objectives (accuracy and robustness). In order to approximate the Pareto-front of different variants of adversarial and shared adversarial training (sharedness $s\in\{1,8,64\}$), we conducted runs for a range of attack parameters: maximum perturbation strength $\varepsilon\in\{2,4,6,8,10,14,18,22,26\}$ and $\sigma\in\{0.3,0.5,0.7,0.9\}$ (controlling the trade-off between expected and adversarial risk). Model fine-tuning was performed with 65 epochs of SGD with batch-size 128, momentum $0.9$, initial learning rate of $0.0025$ and also performed 4 steps of PGD with step-size $\alpha_{k}=0.5\varepsilon$ for each mini-batch. Here, the learning rate was annealed after 50 epochs by a factor of 10.

Figure 3 (left) shows the resulting Pareto fronts of different sharedness values (entries are provided in Table A1 in supplementary material). While sharedness $s=1$ (standard adversarial training) and $s=8$ perform similarly, $s=64$ strictly dominates the other two settings. Without any loss on accuracy, a robustness of $\varepsilon_{uni}(\delta=0.75)=22.7$ can be achieved, and if one accepts an accuracy of $90\%$, a robustness of $\varepsilon_{uni}(\delta=0.75)=44.1$ is obtainable. This corresponds to nearly three times the robustness of the undefended model while accuracy only drops by less than $3.5\%$. We would also like to note that standard adversarial training is surprisingly effective in defending against universal perturbations and achieves a robustness that is smaller by approximately 5 than $s=64$ at the same level of accuracy on unperturbed data. These findings suggest that increasing sharedness results in increased robustness. We found in preliminary experiments that this effect is strong for small $s$ but has diminishing returns for sharedness beyond $s=64$.

We also evaluated the defenses against universal perturbations proposed by Moosavi-Dezfooli et al. [29] and Perolat et al. [37] (please refer to Section A.3 in the supplementary material for details). It can be seen in Figure 3 (left) that these defenses are strictly dominated by all variants of (shared) adversarial training. In terms of computation, shared adversarial training required 189s (the same compute time as required by standard adversarial training) while the defense [29] required 3118s, and the defense [37] required 3840s per epoch on average. The proposed method thus outperforms the baseline defenses both in terms of computation and with regard to the robustness-accuracy trade-off.

Figure 3 (right) shows the Pareto front of the same models when attacked by the DeepFool-based method for generating universal perturbations [29]. In this case, the robustness is computed for a fixed perturbation magnitude $\varepsilon_{uni}=20$ and the accuracy $\delta$ under this perturbation is reported. The qualitative results are the same as for an S-PGD attack: the Pareto-front of adversarial training (s=1) clearly dominates the results achieved by the defense proposed in [29]. Moreover, shared adversarial training with s=64 dominates standard adversarial training and the defense proposed by Perolat et al. [37]. This indicates that the increased robustness by shared adversarial training is not specific to the way the attacker generates universal perturbations. An illustration of the universal perturbations on this dataset is given in Section A.4 in the supplementary material.

6.2 Experiment on a Subset of ImageNet

We extend our experiments to a subset of ImageNet [9], which has more classes and higher resolution inputs than CIFAR10. Please refer to Section A.5 in the supplementary material for details on the selection of this subset. Similar to CIFAR10, we evaluate the robustness using stochastic PGD but generate perturbations on the training set with mini-batches of size $\tilde{m}_{k}=10,000$ and evaluate on the total validation set. We used $b=10$ binary search iterations, $K=20$ S-PGD iterations, and the step-size schedule values $\gamma=0.975$ and $\beta=4$. We pre-trained wide residual network WRN-50-2-bottleneck [49] on this dataset with ERM using SGD for 100 epochs along with initial learning rate 0.1 and reduced it by a factor of 10 after every 30 epochs. We have obtained a top-1 accuracy of 77.57% on unperturbed validation data and a robustness against universal perturbations of $\varepsilon_{uni}(\delta=0.75)=8.4$.

We approximate the Pareto front of adversarial and shared adversarial training with sharedness $s\in\{1,32\}$ and different $\varepsilon\in\{2,4,6,8,10,14,18,22,26\}$ and $\sigma\in\{0.5,1.0\}$. We performed 5 steps of PGD with step-size $\alpha_{k}=0.4\varepsilon$. The model was fine-tuned for 30 epochs of SGD with batch-size 128, momentum term $0.9$, weight decay $5e^{-5}$, an initial learning rate of $0.01$ that was reduced by a factor of 10 after 20 epochs and also performed 5 steps of PGD with step-size $\alpha_{k}=0.4\varepsilon$ for each mini-batch.

Figure 4 compares the Pareto front of shared adversarial training with $s=32$ and standard adversarial training $s=1$ (entries are provided in Table A2 in the supplementary material). It can be clearly seen that shared adversarial training increases the robustness from $\varepsilon_{uni}(\delta=0.75)=8.4$ to $15.0$ without any loss of accuracy. Moreover, shared adversarial training also dominates standard adversarial training for a target accuracy between 67%-74%, which corresponds to the sweet spot as a small loss in accuracy allows a large increase in robustness. The point with accuracy 72.74% and robustness $\varepsilon_{uni}(\delta=0.75)=25.64$ (obtained at $s=32,\varepsilon=10,\sigma=1.0$) can be considered a good trade-off as accuracy drops by only 5% while robustness increases by a factor of 3, which results in clearly perceptible perturbations as shown in the top row of Figure 1 and Section A.6. Moreover, (shared) adversarial training also increases the entropy of the predicted class distribution for successful untargeted perturbations substantially (see Section A.7).

6.3 Semantic Image Segmentation

The results from above experiments show that shared adversarial training improves robustness against universal perturbations on image classification tasks where the adversary aims to fool the classifier’s single decision on an input. In this section, we investigate our method against adversaries in a dense prediction task (semantic image segmentation), where the adversary aims at fooling the classifier on many decisions. To our knowledge, this is the first work to scale defenses based on adversarial training to this task.

We evaluate the proposed method on the Cityscapes dataset [8]. For computational reasons, all images and labels were downsampled from a resolution of $2048\times 1024$ to $1024\times 512$ pixels, where for images a bilinear interpolation and for labels a nearest-neighbor approach was used for downsampling. We pretrained the FCN-8 network architecture [24] on the whole training set of $2975$ images and achieved 49.3% class-wise intersection-over-union (IoU) on the validation set of $500$ images. Note that this IoU is relatively low because of downsampling the images.

We follow the experimental setup of Metzen et al. [27] which performed a targeted attack with a fixed target scene (monchengladbach_000000_026602_gtFine). They demonstrated that the desired target segmentation can be achieved despite the fact that the original scene has nothing in common with the target scene. We use the same target scene and consider this targeted attack successful if the average pixel-wise accuracy between the prediction on the perturbed images and the target segmentation exceeds $\delta=0.95$. For evaluating robustness, we generate $f_{uni}$ using stochastic PGD with mini-batches from the validation set of size $\tilde{m}_{k}=5$ and tested on $16$ samples from test set. We used $b=10$ binary search iterations, $K=200$ S-PGD iterations, the step-size schedule values $\gamma=0.99$ and $\beta=2$, and did not employ loss thresholding for targeted attacks. We find a universal perturbation that upper bounds the robustness of the model to $\varepsilon_{uni}(\delta=0.95)\leq 19.92$.

We fine-tuned this model with adversarial and shared adversarial training. Since approximating the entire Pareto front of both methods would be computationally very expensive, we instead selected a target performance on unperturbed data of roughly 45% IoU (no more than $5\%$ worse than the undefended model). The following two settings achieved this target performance (see Figure 5 left): adversarial training with $\varepsilon=8$ and $\sigma=0.5$ and shared adversarial training for sharedness $s=5$, $\varepsilon=30$, and $\sigma=0.7$. The finetuning was performed for 20 epochs using Adam with batch-size 5 and a learning rate of $0.0001$ that was annealed after 15 epochs to $0.00001$. As heap adversary, we performed 5 steps of untargeted PGD with step-size $\alpha_{k}=0.4\varepsilon$.

While both methods achieved very similar performance on unperturbed data, they show very different robustness against adversarial and universal perturbations (see Figure 5): standard adversarial training largely increases robustness against adversarial perturbations to $\varepsilon_{adv}(\delta=0.95)\leq 11$, an increase by a factor of 4 compared to the undefended model. Shared adversarial training is less effective against adversarial perturbations, its robustness is upper bounded by $\varepsilon_{adv}(\delta=0.95)\leq 5.9$. However, shared adversarial training is more effective against targeted universal perturbations with an upper bound on robustness of $\varepsilon_{uni}(\delta=0.95)\leq 111.7$, while adversarial training reaches $\varepsilon_{uni}(\delta=0.95)\leq 62.5$.

We also evaluated robustness against untargeted attacks: robustness increased from $\varepsilon_{uni}(\delta=0.95)\leq 8.5$ of the undefended model to $25$ and $47.8$ for the models trained with standard and shared adversarial training respectively. The universal perturbation for the model trained with shared adversarial training clearly shows patterns of the target scene and dominates the original image, which is also depicted in the bottom row of Figure 1. We refer to Section A.8 in the supplementary material for illustrations of targeted and untargeted universal perturbations for different models.

6.4 Discussion

Results shown in Figure 5 indicate that there may exist a trade-off between robustness against image-dependent adversarial perturbations and universal perturbations. Figure 6 illustrates why these two kinds of robustness are not strictly related: adversarial perturbations fool a classifier by both adding structure from the target scene/class222For untargeted attacks, the attacks may choose a target scene/class arbitrarily such that fooling the model becomes as simple as possible. to the image (e.g., vegetation on the middle left part of the image) and by destroying properties of the original scene (e.g., edges of the topmost windowsills). The latter is not possible for universal perturbations since the input images are not known in advance. As also shown in the figure, universal perturbations compensate this by adding stronger patterns of the target scene. Shared perturbations will become more similar to universal perturbations with increasing sharedness since a single shared perturbation has fixed capacity and cannot destroy properties of arbitrarily many input images (even if they are known in advance). Accordingly, shared adversarial training will make the model mostly more robust against perturbations which add new structures and not against perturbations which destroy existing structure. Hence, it results in less robustness against image-specific perturbations (seen in Figure 5 middle). On the other hand, since shared adversarial training focuses on one specific kind of perturbations (those that add structure to the scene), it leads to models that are particularly robust against universal perturbations (shown in Figure 5 right).

7 Conclusion

We have shown that adversarial training is surprisingly effective in defending against universal perturbations. Since adversarial training does not explicitly optimize the trade-off between robustness against universal perturbations and performance on unperturbed data points, it handles this trade-off suboptimally. We have proposed shared adversarial training, which performs adversarial training on a tight upper bound of the universal adversarial risk. We have shown that our method allows achieving high robustness against universal perturbations on image classification tasks at smaller loss of accuracy. The proposed method also scales to semantic segmentation on high resolution images, where compared to adversarial training it achieves higher robustness against universal perturbations at the same level of performance on unperturbed images.

Appendix A Supplementary material

A.1 Threat Model

Here, we specify the capabilities of the adversary since the proposed defense mechanism aims at providing security under a specific threat model. We assume a white-box setting, where the adversary has full information about the model, i.e., it knows network architecture and weights, and can provide arbitrary inputs to the model and observe their corresponding outputs (and loss gradients). Moreover, we assume that the attacker can arbitrarily modify every pixel of the input but aims at keeping the $l_{\infty}$ norm of this perturbation minimal. In the case of a universal perturbation, we assume that the attacker can choose an arbitrary perturbation (while aiming to keep the $l_{\infty}$ norm minimal), but crucially does not know the inputs to which this perturbation will be applied. The adversary, however, has access to data points that have been sampled from the same data distribution as the future inputs.

A.2 Relationship of different sharedness

Provided that the heap adversary finds a perturbation that is sufficiently close to the optimal perturbation of the heap and that heaps are composed hierarchically111Heaps are composed hierarchically when a heap of sharedness $2s$ is always the union of two disjoint heaps of sharedness $s$., we have the following relationship for $s=2^{i}$ (we omit the dependence on $\sigma$, $\mathcal{S}$ and $f_{adv}$/$f_{heap}$/$f_{uni}$ for brevity):

[TABLE]

To see $\tilde{\rho}_{heap}^{(s)}\geq\tilde{\rho}_{heap}^{(2s)}$, let $\xi_{1},\dots,\xi_{d/(2s)}$ be the shared perturbations on the $d/(2s)$ heaps of $\tilde{\rho}_{heap}^{(2s)}$. Let $\xi_{j}$ be the shared perturbation for the $j$-th heap. Then, because of the hierarchical construction of the heaps, this heap is composed of two heaps used in $\tilde{\rho}_{heap}^{(s)}$. Let $j_{1}$ and $j_{2}$ be the corresponding indices of these heaps in $\tilde{\rho}_{heap}^{(s)}$. By setting $\xi_{j_{1}}=\xi_{j_{2}}=\xi_{j}$, we obtain $\tilde{\rho}_{heap}^{(s)}=\tilde{\rho}_{heap}^{(2s)}$.

A.3 Configuration of Baselines for CIFAR10

For the defense proposed by Moosavi-Dezfooli et al. [29], we generated 10 different universal perturbations using the DeepFool-based method for generating universal-perturbations on 10,000 randomly sampled training images, ran 5 epochs of adversarial training with $\sigma=0.5$, and chose the applied perturbation uniform randomly from the precomputed perturbations. After these 5 epochs, the robustness was evaluated. This procedure was iterated five times, which resulted in 5 accuracy-robustness points in Figure 1.

We run the defense proposed by Perolat et al. [37] for 45 epochs (sufficiently long for achieving convergence as evidenced by Figure 4 of Perolat et al. [37]). At the beginning of each episode, we generated one universal perturbation using the DeepFool-based method for generating universal-perturbations on the entire training set. We used $\sigma=0.25$, and chose the applied perturbation uniform randomly from all universal perturbations computed so far. We report the accuracy and robustness at the end of these 45 epochs. We note that even though we did not replicate the exact setup of Perolat et al. [37], we achieve a similar accuracy-robustness trade-off in Figure 3 (right) as the one given in Figure 4 of Perolat et al. [37].

A.4 Illustration of Universal Perturbations on CIFAR10

Figure A1 illustrates the minimal universal perturbation found for sharedness $s=1$ and $s=64$ for $\sigma=0.3$ and $\varepsilon\in\{2,8,14,18,26\}$. Universal perturbations of the undefended model resemble high-frequency noise and are quasi-imperceptible when added to an image. Shared adversarial training increases robustness and the resulting perturbations are more perceptible (for small $\varepsilon$) or even dominate the image: for larger $\varepsilon$, the cat in the figure is completely hidden and the perturbed image could also not be classified correctly by a human. Moreover, the perturbation becomes more structured and even object-like for larger $\varepsilon$. Note that the perturbations shown for $s=1$ also achieve high robustness but for smaller accuracy on clean data than those of shared adversarial training with $s=64$.

A.5 Selection of subset of ImageNet

Since the generation of the Pareto fronts on the entire ImageNet dataset would be computationally very expensive, we restrict the experiment to a subset of ImageNet. We use classes defined in TinyImageNet to filter out the samples from ImageNet dataset. We conducted our experiments on the samples of 200 classes from ImageNet, which results in 258,601 train and 10,000 validation images. Note that we take only the list of classes defined from TinyImageNet and use the data of those classes from ImageNet dataset with original resolution.

A.6 Illustration of Universal Perturbations on ImageNet

We depict the universal perturbations with minimum magnitude on different models that are obtained from settings $\sigma=1.0$, sharedness $s\in\{1,32\}$ and different values of $\varepsilon$ on the subset of ImageNet in Figure A2. It can be clearly seen that both the standard ($s=1$) and shared adversarial training ($s=32$) increase robustness when compared against the undefended model but the latter handles the trade-off between performance on unperturbed data and robustness more gracefully. The universal perturbations become clearly visible on a model hardened with shared adversarial training with only a marginal loss of $5\%$ in top-1 accuracy and perturbations become much smoother for larger $\varepsilon$.

A.7 Predicted Class after Untargeted Universal Perturbations

Figure A3 shows which class is predicted on ImageNet validation data after an untargeted universal perturbation (for the respective model) is added. While the undefended model predicts nearly always the same (wrong) class, the models defended with standard and shared adversarial training have a substantially higher entropy in their predictions. Prior work [18] has also observed that undefended models typically misclassify images perturbed with universal perturbation to the same class even though the attack is untargeted. Based on this observation, they hypothesized that directions in which a classifier is vulnerable to universal perturbations coincide with directions important for correct prediction on unperturbed data. We believe it would be important to re-examine these results for a defended model.

A.8 Attacks on Semantic Image Segmentation

We illustrate universal perturbations for targeted and untargeted attacks on different models in this section. We illustrate the effect of the perturbations on one image; however, the perturbations are not specific for this image. For the model trained with empirical risk minimization, Figure A4 shows a targeted attack and Figure A5 an untargeted attack. For the model trained with adversarial training, Figure A6 shows a targeted attack and Figure A8 an untargeted attack. For the model trained with shared adversarial training, Figure A7 shows a targeted attack and Figure A9 an untargeted attack. We also illustrate the universal perturbations found for different models on targeted attacks in Figure A10.