Evolution Attack On Neural Networks

TL;DR

This paper introduces a black-box attack method on neural networks using evolution algorithms, demonstrating that covariance matrix adaptive evolution strategies are most effective in generating adversarial examples without gradient information.

Contribution

The paper formalizes adversarial example generation as an optimization problem and evaluates various evolution algorithms, highlighting the effectiveness of covariance matrix adaptive strategies.

Findings

Covariance matrix adaptive evolution strategy outperforms other algorithms.

Evolution algorithms can effectively generate adversarial examples in a black-box setting.

Regularizations influence the quality of adversarial examples.

Abstract

Many studies have been done to prove the vulnerability of neural networks to adversarial example. A trained and well-behaved model can be fooled by a visually imperceptible perturbation, i.e., an originally correctly classified image could be misclassified after a slight perturbation. In this paper, we propose a black-box strategy to attack such networks using an evolution algorithm. First, we formalize the generation of an adversarial example into the optimization problem of perturbations that represent the noise added to an original image at each pixel. To solve this optimization problem in a black-box way, we find that an evolution algorithm perfectly meets our requirement since it can work without any gradient information. Therefore, we test various evolution algorithms, including a simple genetic algorithm, a parameter-exploring policy gradient, an OpenAI evolution strategy, and a covariance matrix adaptive evolution strategy. Experimental results show that a covariance matrix adaptive evolution Strategy performs best in this optimization problem. Additionally, we also perform several experiments to explore the effect of different regularizations on improving the quality of an adversarial example.

Tables

Table 1. TABLE I: LeNet Architecture

Layer Type	Parameter	Padding	Activation
Conv	Kernel:3x3_20, Stride=1	SAME	ReLu
MaxPool	2x2
Conv	Kernel:3x3_20, Stride=1	SAME	ReLu
MaxPool	2x2
Flatten
FC	500 (Number of neurons)		ReLu
FC	10 (Number of neurons)		SoftMax

Table 2. TABLE II: ResNet-18 Architecture

Layer Type	Parameter	Padding	Activation
Conv	Kernel:3x3_64, Stride=1	SAME	ReLu
Residual Block	Kernel:3x3_64, Stride=1	SAME	ReLu
Residual Block	Kernel:3x3_128, Stride=1	SAME	ReLu
MaxPool	2x2
Residual Block	Kernel:3x3_256, Stride=2	SAME	ReLu
Residual Block	Kernel:3x3_512, Stride=1	SAME	ReLu
MaxPool	2x2
Conv	Kernel:3x3_512, Stride=1	SAME	ReLu
Flatten
FC	10 (Number of neurons)		SoftMax

Table 3. TABLE III: MNIST Experiment: Performance of Different EAs in Adversarial Attack

	Similarity Evaluation
EA	$L_{1}Norm$	$L_{2}Norm$	$L_{\infty }Norm$	SSIM
GA(25)	34.5883	1.9533	0.3006	0.8037
GA(100)	32.0961	1.7931	0.2671	0.8151
PEPG(25)	38.0039	2.1237	0.2875	0.7914
PEPG(100)	32.4222	1.7693	0.2122	0.8164
OpenAI(25)	35.1598	1.9473	0.2523	0.8034
OpenAI(100)	30.8387	1.6566	0.1841	0.8251
CMA-ES(25)	25.4210	1.4649	0.2556	0.8470
CMA-ES(100)	32.2897	1.8284	0.3024	0.8117

Table 4. TABLE IV: MNIST Experiment: Performance of Different Regularizations on Adversarial Attack

	Similarity Evaluation
EA	$L_{1}Norm$	$L_{2}Norm$	$L_{\infty }Norm$	SSIM
CMA-ES(25)	25.4210	1.4649	0.2556	0.8470
CMA-ES${}_{\left(25\right)-L_2}$	24.2534	1.4062	0.2486	0.8531
CMA-ES${}_{\left(25\right)-L_{\infty }}$	25.5840	1.4610	0.2159	0.8497
CMA-ES(100)	32.2897	1.8284	0.3024	0.8117
CMA-ES${}_{\left(100\right)-L_2}$	31.4301	1.7800	0.2964	0.8153
CMA-ES${}_{\left(100\right)-L_{\infty }}$	25.8740	1.4766	0.2315	0.8484

Table 5. TABLE V: Cifar-10 Experiment: Different Regularizations on Adversarial Attack

	Similarity Evaluation
EA	$L_{1}Norm$	$L_{2}Norm$	$L_{\infty }Norm$	SSIM
CMA-ES(25)	150.8968	3.4296	0.2329	0.7908
CMA-ES${}_{\left(25\right)-L_2}$	150.2518	3.4152	0.2313	0.7921
CMA-ES${}_{\left(25\right)-L_{\infty }}$	150.6225	3.4229	0.2295	0.7915
CMA-ES(100)	124.5613	2.8314	0.1925	0.8370
CMA-ES${}_{\left(100\right)-L_2}$	123.9939	2.8182	0.1908	0.8384
CMA-ES${}_{\left(100\right)-L_{\infty }}$	124.5772	2.8310	0.1884	0.8474

Equations

$$\mathbf{\eta }=ϵsign({\nabla }_{\mathbf{x}}J(\mathbf{\theta },\mathbf{x},y)).$$

$$\mathbf{\eta }=ϵ({\nabla }_{\mathbf{x}}J(\mathbf{\theta },\mathbf{x},y)).$$

$$\begin{array}{rl}\mathcal{N}(\mathbf{m},\mathbf{C})& \sim \mathbf{m}+\mathcal{N}(0,\mathbf{C})\\ & \sim \mathbf{m}+{\mathbf{C}}^{1/2}\mathcal{N}(0,\mathbf{I})\\ & \sim \mathbf{m}+\mathbf{B}\mathbf{D}\underset{\sim \mathcal{N}(0,\mathbf{I})}{\underbrace{{\mathbf{B}}^{\mathsf{T}}\mathcal{N}(0,\mathbf{I})}}\\ & \sim \mathbf{m}+\mathbf{B}\underset{\sim \mathcal{N}(0,{\mathbf{D}}^2)}{\underbrace{\mathbf{D}\mathcal{N}(0,\mathbf{I})}}.\end{array}$$

$${\mathbf{p}}_c^{(g+1)}=(1-c_c){\mathbf{p}}_c^{(g)}+\sqrt{c_c(2-c_c)}\frac{{\mathbf{m}}^{(g+1)}-{\mathbf{m}}^{(g)}}{{\sigma }^{(g)}}.$$

$${\mathbf{C}}^{(g+1)}=(1-c_1){\mathbf{C}}^{(g)}+c_1{\mathbf{p}}_c^{(g+1)}{\mathbf{p}}_c^{(g+1{)}^{\mathsf{T}}}.$$

$$\begin{array}{rl}{\mathbf{C}}^{(g+1)}& =(1-c_{\mu }){\mathbf{C}}^{(g)}+c_{\mu }\frac{1}{{\sigma }^{(g{)}^2}}\\ & =(1-c_{\mu }){\mathbf{C}}^{(g)}+c_{\mu }\sum _{i=1}^{\mu }{w}_i{\mathbf{y}}_{i:\lambda }^{(g+1)}{\mathbf{y}}_{i:\lambda }^{(g+1{)}^{\mathsf{T}}}.\end{array}$$

$${\mathbf{m}}^{(g+1)}={\mathbf{m}}^{(g)}+c_m\sum _{i=1}^{\mu }{w}_i({\mathbf{x}}_{i:\lambda }^{(g+1)}).$$

$$\arg \underset{\mathbf{W}}{\min }\mathbf{D}(\mathbf{Y},\mathbf{F}(\mathbf{X},\mathbf{W})),$$

$$\arg \underset{\mathbf{\eta }}{\min }\mathbf{D}({\mathbf{Y}}^{\prime },\mathbf{F}(\mathbf{X}+\mathbf{\eta },\mathbf{W}))+\beta \parallel \mathbf{\eta }\parallel ,$$

Full text

I Introduction

Currently, deep convolutional neural networks (DCNNs) have become the state-of-art learning model in the computer vision field. Since AlexNet[1] won the ImageNet Large Scale Visual Recognition Competition (ILSVRC) in 2012, an increasing number of researchers have been attracted to DCNNs due to their powerful abilities of learning and feature extraction. Up to now, almost every advanced solution for image problems are based on DCNNs, such as image classification, target detection[2][3], semantic segmentation[4][5], and pose estimation[6]. With the development of computer vision technology, many intelligent applications have emerged, e.g., face recognition for identity authentication and visual perception for intelligent vehicles. Therefore, the security of these intelligent systems has become increasingly more important.

Although DCNNs have shown impressive results in such fields, more attention must be paid to their weaknesses, namely, that they do not constitute a robust model and are vulnerable to adversarial example attacks[7]. For basic image classification problems in computer vision, assuming a trained classifier exists, the purpose of such attacks is to find the perturbations that can make the input image able to fool the classifier. After being perturbed, the image that is originally correctly classified will be misclassified by the classifier. The perturbed image is called an adversarial example. In addition, the perturbations must be as small as possible; otherwise, the abnormal image can be easily found and rejected manually.

The issue in generating a qualified, quality adversarial example is to find the perturbations that meet the following requirements:

• •

Perturbations that lead to the misclassification of an image that is correctly classified.

• •

Perturbations need to be as unperceptive to people as much as possible.

Based on access right to model gradient information, two kinds of attacks exist: white-box attacks and black-box attacks.

A white-box attack is one in which attackers need to know detailed information about the target DCNN model. For example, the efficient attack algorithms fast gradient sign (FGS)[8] and fast gradient value (FGV)[9] require that the gradient information of classifier output for every input pixel element be known. It is implied that these methods will not work without knowledge of the internal architecture of the classifier. Considering an actual situation, it is difficult for an attacker to know such information, so the black-box attack method is much more dangerous than the white-box attack method.

In contrast to a white-box attack, the basic and important requirement for a black-box attack is to generate an adversarial example without detailed information of the classifier system. The only knowledge required is the output vector of the classifier.

To attack a target model without gradient information, we find that the intelligent optimization method called an evolution algorithm (EA)[21], whose primary characteristic, i.e., no demand for gradient information, perfectly meets the requirements of a black-box attack if the problem of adversarial example generation can be transformed into the optimization problem of perturbation. EAs comprise another large field of study, compared to numerical analysis optimization methods like gradient descent and the Newton method, and are closer to space searching. To solve the above optimization problem, we propose an advanced EA named the covariance matrix adaptation evolution strategy (CMA-ES)[20][22][23]. In addition, we test other EAs for comparison, such as a simple genetic algorithms (GA)[16], a parameter-exploring policy gradient (PEPG)[17][18], and the OpenAI evolution strategy (OpenAI ES)[19]. To improve the quality of adversarial examples, we also performed several experiments to explore the effects using different regularizations.

The main work presented in this paper comprises the following:

• •

Formalizing the process of generating an adversarial example.

• •

Successfully generating the adversarial example by our black-box attack strategy. As the most indispensable part, the efficiency of the EA directly determines the quality of the adversarial example. Therefore, we test different EAs, such as a GA, PEPG, OpenAI ES, and CMA-ES in our strategy.

• •

Considering that our method can generate an adversarial example of 100%, the quality, i.e., the similarity between the original and perturbed images, is the most valuable evaluation index for our attack method. Thus, we evaluate the adversarial examples generated by different EAs. In addition, we also try different regularizations of our optimization target to reduce the perturbations.

II Related Work

Although DCNNs have been proven to be a powerful learning model, Szegedy et al.[7] first found that a hardly perceptible perturbation would lead to misclassification of the trained Network. Goodfellow et al. [8] applied the FGS algorithm to generate an adversarial example and explained that the linear nature of DCNNs was responsible for its vulnerability:

[TABLE]

Rozsa et al.[9] improved the FGS and proposed the FGV algorithm:

[TABLE]

Moosavi-Dezfooli et al. [10] suggested an iteration method named DeepFool to minimize the perturbation $\bm{\eta}$, which is sufficient to change the predicted result. The existence of universal perturbations was shown in [12]. Baluja et al.[14] proposed the adversarial transformation network, which is similar to an autoencoder, to generate an adversarial example. Xie et al.[13] developed an adversarial example for semantic segmentation and object detection.

The above works all concern a white-box attack. Considering a black-box attack is more dangerous in the real world, Papernot et al.[11] proposed the black-box attack strategy, in which the substitute model needs to be trained using the outputs of the original model as labels. Thus, an adversarial example based on detailed information of the new substitute model can also fool the original model.

For adversarial defense, Goodfellow et al.[8] found that adversarial training could reduce network overfitting and, in turn, reinforce the robustness. Papernot et al.[15] applied defensive distillation to improve the robustness of the network. Lee et al.[24] tried to use a generative adversarial network (GAN) to train a robust classifier. Its generator is supposed to generate perturbations and its classifier is supposed to correctly classify both the original and perturbed images.

III CMA-ES

Below, we introduce the CMA-ES, which, as one of the most advanced EAs, plays the most significant role in the black-box attack method proposed in this paper.

Before the official introduction of CMA-ES, we briefly discuss the basic characteristics and history of EAs. Different from the numerical analysis optimization methods, such as the gradient descent or Newton methods, an EA requires fewer assumptions and can work without the gradient information. Therefore, it can be seen as the black-box optimization method.

In the beginning, EA was developed from the central idea of Darwin’s theory of evolution, i.e., survival of the fittest. Assuming there are large numbers of populations, after natural selection, the survival populations, called elite populations, recombine with each other to generate the new generation of populations that are supposed to have better survival ability then the previous generation. In an EA, the survival ability can be described as a fitness function, i.e., one that is equal to the loss function in an optimization problem. The above introduction can be seen as the basic EA process. With the passage of time, increasingly more innovations have occurred in EAs, including various methods of population generation, selection, and recombination.

Simple EAs behave well in low-dimensional optimization problems, but nevertheless show poor ability when facing high-dimensional optimization problems. Compared to gradient descent, an EA is more like the space searching method, which means it must face the ”dimension explosion” problem with the increase in problem dimensionality.

We submit that CMA-ES can improve the optimization efficiency to some extent. Compared to other EAs, the largest difference in CMA-ES is that it introduces a priori knowledge of problem distribution. Unlike GA and other simple EAs that treat different dimensions independently, CMA-ES regards the optimization problem as a multivariate normal distribution defined by the mean vector and covariance matrix. Every generation of populations is sampled from the current multivariate normal distribution, further, selection and recombination are designed to update the mean vector and covariance matrix to make the distribution closer to the local optimal point. The multivariate normal distribution $\mathcal{N}(\bm{m},\bm{C})$ is written in (3):

[TABLE]

The covariance matrix $\bm{C}\in\mathbb{R}^{n*n}$ is symmetric and positive definite, so $\bm{C}=\bm{B}\bm{D}^{2}\bm{B}^{\mathsf{T}}$ is the eigen-decomposition of $\bm{C}$, and then $\bm{C}^{1/2}=\bm{B}\bm{D}\bm{B}^{\mathsf{T}}$.

$\mathcal{N}(\bm{0},\bm{I})$ produces a spherical(isotropic) distribution, as in the left-hand panel of Fig. 2. $\bm{D}$ scales the spherical distribution within the coordinate axes as in the middle panel of Fig. 2. $\bm{D}\mathcal{N}(\bm{0},\bm{I})\sim\mathcal{N}(\bm{0},\bm{D}^{2})$ has n independent components. The matrix $\bm{D}$ can be interpreted as an individual step-size matrix and its diagonal entries are the standard deviations of the components. $\bm{B}$ defines a new orientation for the ellipsoid, where the new principal axes of the ellipsoid correspond to the columns of $\bm{B}$.

The transformation in (3) helps sample the population from the current distribution in a computer considering that $\mathcal{N}(\bm{0},\bm{I})$ actually represents the independent normal distribution in every dimension.

Every element in the covariance matrix is the covariance between every two dimensions. Considering the real complex optimization problem is always highly coupled in different dimensions, and the covariance matrix can represent such a complex relationship in some ways if the iteration update is effective. The effective measure of the update will increase the variance along the efficient searching directions. In fact, thinking about the coordinate transformation of a vector in a new coordinate system in principal component analysis (PCA), the main directions represented in the covariance matrix are equivalent to the base vector in the new coordinate system.

The main selection and recombination formed in the CMA-ES include two kinds of methods to adapt the covariance matrix and one kind of method to update the mean vector after selection.

III-A Rank-1 Update

Roughly speaking, a rank-1 update is a historical accumulation of the trajectory of the elite populations’ mean vector $\bm{m}\in\mathbb{R}^{n}$. Thus, CMA-ES defines the evolution path to record the trajectory of $\bm{m}$. In fact, $\bm{p}_{c}$ is the weighted average of $\frac{\bm{m}^{(g+1)}-m^{(g)}}{\sigma}$ in every iteration:

[TABLE]

$\bm{p}_{c}^{(g)}\in\mathbb{R}^{n}$ is the evolution path at generation $g$ and $c_{c}\leq 1$ the balance coefficient for historical accumulation and current generation.

The rank-1 update for the covariance matrix is expressed as

[TABLE]

III-B Rank-$\mu$-Update

The rank-$\mu$ update plays an important role when population size increases. It is actually the weighted maximum-likelihood estimation for the covariance matrix $\bm{C}$:

[TABLE]

where $c_{\mu}$ is the learning rate for updating the covariance matrix, $w_{1,\ldots,\mu}\in\mathbb{R}$, $\sum_{i}w_{i}=1$ and $w_{1}\geq\ldots\geq w_{\mu}\geq 0$, $\bm{y}_{i:\lambda}^{(g+1)}=(\bm{x}_{i:\lambda}^{(g+1)}-\bm{m}^{(g)})/\sigma^{(g)}$

III-C Mean Vector Update

The mean vector update is expressed as

[TABLE]

where $c_{m}$ is the learning rate for updating the mean vector.

We only introduce the above basic concept of CMA-ES in this paper considering that CMA-ES is a complicated subject. It is a necessary part of our black-box attack strategy as it can optimize without gradient information. Further, CMA-ES also involves the control of the learning-step size. The detailed analysis of CMA-ES can be found in [20].

IV Black-Box Attack Strategy

Here, we introduce our black-box attack strategy.

To a classifier, the process of attack can be seen as the generation of an adversarial example. A black-box strategy means we must attack without any internal architecture information about the target model. The only accessible information about the target model is its output vector. The aim of attack is to find the perturbations that would lead to the misclassification of the target model. Furthermore, we can choose the attack target, i.e., the desired classification result after the input image is perturbed, and our strategy is able to manipulate the misclassification result on any class desired.

First, we must determine the attack target and the one-hot vector that could represent the attack target. Then, the formalization must be done to set the optimization problem of the perturbation.

The training step for the neural network can serve as a comparison for such an optimization problem.The supervised learning for the neural network will find the suitable network weight to minimize the loss function (like cross-entropy) at a given input and corresponding label as it is described in the following equation:

[TABLE]

where $\bm{X}$ is the input image, $\bm{W}$ are the weights of the neural network, $\bm{Y}\in\mathbb{R}^{n}$ corresponds to the one hot label vector for input $\bm{X}$, $\bm{F}()\in\mathbb{R}^{n}$ is the classifier function, i.e., network, $\bm{D}()\in\mathbb{R}$ is the loss function for supervised learning, e.g., cross-entropy.

For an adversarial attack, the network is trained, suggesting that $\bm{W}$ is fixed and the parameters that must be optimized are the perturbations $\bm{\eta}$. If we set the adversarial label $\bm{Y}^{\prime}$, which represents the attack target for the corresponding input $\bm{X}$, the minimization process will finally find the perturbation that enables the input image $\bm{X}$ to fool the classifier:

[TABLE]

where $\bm{\eta}\in\mathbb{R}^{w*h*c}$ is the noise of the original input image and $\parallel.\parallel$ is the norm of the tensor, e.g., $L_{2},L_{\infty}$. To limit the perturbations, $\beta$ is the balance coefficient in multi-objective optimization.

We have thus set the optimization objective. The next step is to use the EA to solve this problem. Additionally, we need to set the terminal conditions for evolution iteration, as expressed in Algorithm 2.

V Experiment

To test our black-box attack strategy, we require a trained network, and we therefore trained two classifiers that meet the baseline criteria: 1) LeNet on MNIST, which can achieve 98.9% accuracy on the test set, and 2) simple ResNet$-$18 on Cifar$-$10, which can achieve 90.1% accuracy on the test set. The architectures of these networks are described in Tables I and  II, respectively.

In the MNIST experiment, we tested different EAs, including a simple GA, PEPG, OpenAI ES, and CMA-ES. We also tested different population sizes, i.e., 25 and 100. The elite populations ratio was set to 50%, initial mean vector to $\bm{0}^{w*h*c}$, and learning step size to 0.05. For an adversarial target, we chose the second-most likely class predicted by the classifier as the original input image. In this case, the evolution iteration would keep attacking until the desired misclassification occurred. For regularization in (9), we set $\beta$ as 1.

These terminal conditions ensure that our attack method could generate an adversarial example in 100% of the cases. In other words, the quality of the adversarial example is the most important issue for evaluating the strategy. We define the quality of the adversarial example as the similarity between the original image and adversarial example. We therefore measured the ${L_{1}Norm}$, ${L_{2}Norm}$, and $L_{\infty}Norm$ of perturbations. Additionally, we used the structural similarity index (SSIM) to evaluate the two images’ similarity.

As a realization of the structural similarity theory, the SSIM defines the structural information from the perspective of image composition independent of brightness and contrast, reflecting the properties of the object structure in the scene. The mean is used as the estimate of the brightness, the standard deviation as the estimate of the contrast, and the covariance as a measure of the degree of structural similarity. To a certain extent, the SSIM can reflect the difference between the original and perturbed images in human visual perception. SSIM values range from 0 to 1, where 1 means that two images are exactly the same.

From Table. III, it can be seen that a simple GA, PEPG, and OpenAI-ES improve their optimization effect with increased population size. However, CMA-ES performed better in population sizes of 25 than in population sizes of 100 without any regularization. We think the reason for this phenomenon is the recombination method in the algorithm. For simple optimization problems, the increase of population size leads to an increase in the number of elite populations, which brings some noise to the distribution update. Since images in MNIST have 28281 pixels and images in Cifar$-$10 have 32323 pixels, it is not suitable to compare the $L_{1}Norm$ and $L_{2}Norm$ of the different datasets. We only observe the performance of different population sizes in the same dataset. In Table. V, for the Cifar$-$10 experiment, it can be seen that, for large population sizes, CMA-ES performs better. Owing to the larger number of dimensions in optimization, the opposite result is shown for the MNIST experiment. Therefore, one cannot just say which population size is better, but performance is determined by the actual situation. For low-dimensional problems, large population size does not guarantee better performance. For high-dimensional problems, the optimization result may be improved because of more populations. The quantitative relationship between dimension and population size merits more attention. Further, another factor is the weighted method of elite populations. A smoother weighted method means more global searching, in other words, the approach may not be efficient enough. In our attack method, the more efficient optimization method means a higher-quality adversarial image. Thus, more time is still needed to explore how to improve the optimization method in such situations.

Another issue worth discussing is regularization. From Table. IV, it can be seen that additional regularization indeed improves the performances of our attack method. The optimization difficulty increases with added regularization, and turns out to be the multi-objective optimization problem. In such situations, we think a large-populations-size CMA-ES could work better. Experimental results show that a large-population-size CMA-ES exhibits more obviously improved performance, while a small-population-size CMA-ES still shows better problem-solving ability. In Table. V, for the Cifar$-$10 experiment, the benefits from different regularizations are not obvious enough to say which regularization is better. We regard the SSIM as the most important index in our evaluation. Our thinking is, if large population size can bring a more direct searching direction in the multi-objective optimization problem, then the $L_{\infty}Norm$ may be the better choice to limit perturbations, because in both the MNIST and Cifar$-$10 experiments, $L_{\infty}Norm$ helps more than $L_{2}Norm$ in large-population-size CMA-ES. However, we are not very confident in this conclusion, and more detailed experiments should be conducted to vet this idea.

The hyper-parameter of CMA-ES, as the most indispensable and important part of our evolution attack method, will directly influence the quality of an adversarial attack. As with the evolution algorithm, it is impossible to say which hyper-parameter is definitely the best. However, our experiment fortunately shows that it is perfectly suitable for our evolution attack.

VI Acknowledgements

This work is supported by the Project supported by the National Key Research and Development Program (Grant no. 2016YFB0100901), the Shanghai Municipal Commission of Economy and Informatization (Grant no. 2018RGZN02050), and the Science and Technology Commission of Shanghai (Grant no. 17DZ1100202; 16DZ1100700).