On Output Activation Functions for Adversarial Losses: A Theoretical Analysis via Variational Divergence Minimization and An Empirical Study on MNIST Classification

TL;DR

This paper provides a theoretical analysis of output activation functions for adversarial losses using variational divergence minimization and empirically evaluates 168 combinations on MNIST, offering guidance for designing effective adversarial losses.

Contribution

It derives necessary and sufficient conditions for well-behaved adversarial losses and introduces a framework to empirically compare various activation and regularization combinations.

Findings

No single best combination of activation and regularization across all settings.

A large class of theoretically valid adversarial losses identified.

Empirical evaluation on MNIST highlights the variability in performance.

Abstract

Recent years have seen adversarial losses been applied to many fields. Their applications extend beyond the originally proposed generative modeling to conditional generative and discriminative settings. While prior work has proposed various output activation functions and regularization approaches, some open questions still remain unanswered. In this paper, we aim to study the following two research questions: 1) What types of output activation functions form a well-behaved adversarial loss? 2) How different combinations of output activation functions and regularization approaches perform empirically against one another? To answer the first question, we adopt the perspective of variational divergence minimization and consider an adversarial loss well-behaved if it behaves as a divergence-like measure between the data and model distributions. Using a generalized formulation for adversarial losses, we derive the necessary and sufficient conditions of a well-behaved adversarial loss. Our analysis reveals a large class of theoretically valid adversarial losses. For the second question, we propose a simple comparative framework for adversarial losses using discriminative adversarial networks. The proposed framework allows us to efficiently evaluate adversarial losses using a standard evaluation metric such as the classification accuracy. With the proposed framework, we evaluate a comprehensive set of 168 combinations of twelve output activation functions and fourteen regularization approaches on the handwritten digit classification problem to decouple their effects. Our empirical findings suggest that there is no single winning combination of output activation functions and regularization approaches across all settings. Our theoretical and empirical results may together serve as a reference for choosing or designing adversarial losses in future research.

Figures

Tables

Table 1. Table 1: Distribution p 𝒙 ^ subscript 𝑝 ^ 𝒙 p_{\hat{\bm{x}}} and function R 𝑅 R in Equation   11 for common gradient penalties, where k ∈ ℝ 𝑘 ℝ k\in\mathbb{R} is the Lipschitz constant and c ∈ ℝ 𝑐 ℝ c\in\mathbb{R} is a hyperparameter.

	$p_{\widehat{𝒙}}$	$R\left(r\right)$
Two-side coupled gradient penalty (Gulrajani et al., 2017)	$P_d+U[0,1]\left(P_g-P_d\right)$	${\left(r-k\right)}^2$
One-side coupled gradient penalty (Gulrajani et al., 2017)	$P_d+U[0,1]\left(P_g-P_d\right)$	$\max (r,k)$
Two-side local gradient penalty (Kodali et al., 2017)	$P_d+cN(0,I)$	${\left(r-k\right)}^2$
One-side local gradient penalty (Kodali et al., 2017)	$P_d+cN(0,I)$	$\max (r,k)$
R1 gradient penalty (Mescheder et al., 2018)	$P_d$	$r^2/2$
R2 gradient penalty (Mescheder et al., 2018)	$P_g$	$r^2/2$

Table 2. Table 2: Output activation functions for the adversarial losses discussed in this paper (see Equation   12 and Equation   13 for the definitions of f 𝑓 f , g 𝑔 g and h ℎ h ). In the rightmost column, y ∗ superscript 𝑦 y^{*} denotes the common root of f ​ ( y ) = g ​ ( y ) 𝑓 𝑦 𝑔 𝑦 f(y)=g(y) and f ′ ​ ( y ) = − g ′ ​ ( y ) ≠ 0 superscript 𝑓 ′ 𝑦 superscript 𝑔 ′ 𝑦 0 f^{\prime}(y)=-g^{\prime}(y)\neq 0 (see Section   4.4 for more details).

	$f$	$g$	$h$	$y^{\ast }$
Classic minimax (2014)	$-\log \left(1+e^{-y}\right)$	$-y-\log \left(1+e^{-y}\right)$	$-y-\log \left(1+e^{-y}\right)$	$0$
Classic nonsaturating (2014)	$-\log \left(1+e^{-y}\right)$	$-y-\log \left(1+e^{-y}\right)$	$\log \left(1+e^{-y}\right)$	$0$
Wasserstein (2017)	$y$	$-y$	$-y$	$0$
Least squares (2017)	$-{\left(1-y\right)}^2$	$-y^2$	${\left(1-y\right)}^2$	$1/2$
Hinge (2017; 2017)	$\min (0,y-1)$	$\min (0,-1-y)$	$-y$	$0$
Absolute	$-\left|1-y\right|$	$-\left|y\right|$	$\left|1-y\right|$	$1/2$
Asymmetric	$-\left|y\right|$	$-y$	$-y$	$0$

Table 3. Table 3: Network architectures for the (a) generator G 𝐺 G and (b) discriminator D 𝐷 D used in our experiments.

Layer type	Filters/neurons	Kernel	Stride
Convolutional	32	$3\times 3$	$3\times 3$
Convolutional	64	$3\times 3$	$3\times 3$
Max pooling	-	$2\times 2$	$2\times 2$
Dense	128	-	-
Dense	10	-	-

Table 4. Table 4: Error rates (%) for different combinations of output activation functions and regularization approaches. Mean values over ten runs and 95% confidence intervals are reported. Underlined and bold fonts indicate the lowest error rates per column and per row, respectively. (Abbreviations: GP—gradient penalty, SN—spectral normalization, TCGP—two-side coupled gradient penalty, TLGP—two-side local gradient penalty, OCGP—one-side coupled gradient penalty, OLGP—one-side local gradient penalty.)

	Unregularized	TCGP	TLGP	R1 GP	R2 GP
Classic minimax (2014)	9.11 $\pm$  0.39	5.65 $\pm$ 0.17	5.42 $\pm$ 0.11	19.01 $\pm$  2.31	12.91 $\pm$  0.70
Classic nonsaturating (2014)	26.83 $\pm$  4.44	5.64 $\pm$ 0.14	5.56 $\pm$ 0.19	14.67 $\pm$  3.01	13.80 $\pm$  1.98
Classic linear	17.38 $\pm$  3.20	5.66 $\pm$ 0.22	5.55 $\pm$ 0.10	18.49 $\pm$  3.42	14.92 $\pm$  3.22
Hinge minimax	5.57 $\pm$  0.16	4.83 $\pm$ 0.21	4.88 $\pm$ 0.15	7.31 $\pm$  0.92	9.49 $\pm$  3.28
Hinge nonsaturating	37.55 $\pm$ 12.53	5.00 $\pm$ 0.15	4.97 $\pm$ 0.15	7.34 $\pm$  1.13	7.54 $\pm$  0.81
Hinge linear (2017; 2017)	11.50 $\pm$  3.30	5.01 $\pm$ 0.16	4.89 $\pm$ 0.11	8.96 $\pm$  2.20	7.71 $\pm$  1.13
Wasserstein (2017)	7.69 $\pm$  0.20	5.04 $\pm$ 0.12	4.92 $\pm$ 0.14	13.89 $\pm$ 12.79	7.25 $\pm$  0.74
Least squares (2017)	7.15 $\pm$  0.29	7.27 $\pm$ 0.27	6.70 $\pm$ 0.27	30.12 $\pm$ 17.62	32.44 $\pm$ 13.05
Relativistic average (2019)	90.20 $\pm$  0.00	5.25 $\pm$ 0.15	5.01 $\pm$ 0.19	8.00 $\pm$  1.01	8.75 $\pm$  3.61
Relativistic average hinge (2019)	52.01 $\pm$  5.81	8.28 $\pm$ 6.36	4.71 $\pm$ 0.07	8.39 $\pm$  1.19	7.67 $\pm$  1.13
Absolute	6.69 $\pm$  0.15	5.23 $\pm$ 0.18	5.20 $\pm$ 0.16	8.01 $\pm$  1.21	6.64 $\pm$  0.32
Asymmetric	7.81 $\pm$  0.17	4.77 $\pm$ 0.21	4.94 $\pm$ 0.09	8.79 $\pm$  1.97	7.33 $\pm$  0.63

Table 5. Table 5: Error rates (%) for the nonsaturating, the Wasserstein and hinge losses along with their ω 𝜔 \omega -weighted versions (see Equation   22 for the definition). Mean values and 95% confidence intervals are reported.

	$\omega =0.5$	$\omega =0.9$	Original	$\omega =1.1$	$\omega =2.0$
Nonsaturating	8.47 $\pm$ 0.22	8.96 $\pm$ 0.39	8.25 $\pm$ 0.22	8.62 $\pm$ 0.28	9.18 $\pm$ 0.58
Wasserstein	73.16 $\pm$ 3.94	57.66 $\pm$ 3.18	5.89 $\pm$ 0.16	60.30 $\pm$ 4.72	69.54 $\pm$ 3.33
Hinge	15.20 $\pm$ 1.52	8.94 $\pm$ 0.54	6.59 $\pm$ 0.19	8.02 $\pm$ 0.22	11.87 $\pm$ 0.53

Table 6. Table 6: The corresponding output activation functions for different f 𝑓 f -GANs.

Name	$\hat{f}$	$g\left(={\hat{f}}^{\ast }\right)$	$g^{\prime }$	$y^{\ast }$
Total variation	$\left|y-1\right|/2$	$-y$	$-1$	$0$
Kullback–Leibler (KL)	$y\log y$	$-e^{y-1}$	$-e^{y-1}$	-
Reverse KL	$-\log y$	$1+\log \left(-y\right)$	$1/y$	-
Pearson ${\chi }^2$	${\left(1-y\right)}^2$	$-y-y^2/4$	$-1-y/2$	$0$
Neyman ${\chi }^2$	${\left(1-y\right)}^2/y$	$-2+2\sqrt{1-y}$	$-1/\sqrt{1-y}$	$0$
Squared Hellinger	${\left(1-\sqrt{y}\right)}^2$	$-y/\left(1-y\right)$	$-1/{\left(1-y\right)}^2$	0
Jensen-Shannon	$-\left(1+y\right)\log \left(\left(1+y\right)/2\right)+y\log y$	$\log \left(2-e^y\right)$	$-e^y/\left(2-e^y\right)$	0

Table 7. Table 7: The corresponding ψ 𝜓 \psi functions for different f 𝑓 f -GANs and their minimum points.

Name	$\psi \left(\gamma \right)$	${\arg \min }_{\gamma }\psi \left(\gamma \right)$
Total variation	$\left|\gamma -1/2\right|$	$1/2$
Kullback–Leibler (KL)	$\gamma \log \left(\gamma /\left(1-\gamma \right)\right)$	$\approx 0.2178$
Reverse KL	$\left(1-\gamma \right)\log \left(\left(1-\gamma \right)/\gamma \right)$	$\approx 0.7822$
Pearson ${\chi }^2$	${\left(1-2\gamma \right)}^2/\left(1-\gamma \right)$	$1/2$
Neyman ${\chi }^2$	${\left(1-2\gamma \right)}^2/\gamma$	$1/2$
Squared Hellinger	$1-2\sqrt{\gamma \left(1-\gamma \right)}$	$1/2$
Jensen-Shannon	$\log 2+\gamma \log \gamma +\left(1-\gamma \right)\log \left(1-\gamma \right)$	$1/2$

Equations

$$\underset{G}{\min }\underset{D}{\max }$$

$$D_f(P_d,P_g)\ge \underset{D\in \mathcal{D}}{\sup }{\mathbb{E}}_{\mathbf{x}\sim P_d}[D(\mathbf{x})]-{\mathbb{E}}_{\tilde{\mathbf{x}}\sim P_g}[f^{\ast }(D(\tilde{\mathbf{x}}))],$$

$$\underset{G}{\min }\underset{D}{\max }$$

$$W(P_d,P_g)=\underset{||f|{|}_L\le 1}{\sup }{\mathbb{E}}_{\mathbf{x}\sim P_d}[f(\mathbf{x})]-{\mathbb{E}}_{\tilde{\mathbf{x}}\sim P_g}[f(\tilde{\mathbf{x}})],$$

$$\underset{G}{\min }\underset{D\in \mathcal{D}}{\max }{\mathbb{E}}_{\mathbf{x}\sim P_d}[f(\mathbf{x})]-{\mathbb{E}}_{\tilde{\mathbf{x}}\sim P_g}[f(\tilde{\mathbf{x}})],$$

$$\underset{D}{\min }$$

$$\underset{G}{\min }$$

$$\underset{D}{\min }$$

$$\underset{G}{\min }$$

$$\lambda {\mathbb{E}}_{\hat{\mathbf{x}}\sim p_{\hat{\mathbf{x}}}}[R(||{\nabla }_{\hat{\mathbf{x}}}D(\hat{\mathbf{x}})||)],$$

$$\underset{D}{\max }$$

$$\underset{G}{\min }$$

$$\underset{G}{\min }\underset{D}{\max }$$

$$L_G$$

$$L_G=\underset{D}{\max }{\int }_{\mathbf{x}}{P}_d(\mathbf{x})f(D(\mathbf{x}))+P_g(\mathbf{x})g(D(\mathbf{x}))d\mathbf{x},$$

$$L_{G}=\int_{\bm{x}}\max_{D}\;\Big{(}P_{d}(\bm{x})\,f(D(\bm{x}))+P_{g}(\bm{x})\,g(D(\bm{x}))\Big{)}\,d\bm{x}\,,$$

$$L_{G}=\int_{\bm{x}}\big{(}P_{d}(\bm{x})+P_{g}(\bm{x})\big{)}\max_{D}\left(\frac{P_{d}(\bm{x})\,f(D(\bm{x}))}{P_{d}(\bm{x})+P_{g}(\bm{x})}+\frac{P_{g}(\bm{x})\,g(D(\bm{x}))}{P_{d}(\bm{x})+P_{g}(\bm{x})}\right)\,d\bm{x}\,.$$

$$L_{G}=\int_{\bm{x}}\big{(}P_{d}(\bm{x})+P_{g}(\bm{x})\big{)}\,\max_{y}\;\gamma\,f(y)+(1-\gamma)\,g(y)\,d\bm{x}\,.$$

$$\Psi (\gamma ,y)$$

$$\psi (\gamma )$$

$$\underset{D}{\max }{\mathbb{E}}_{\mathbf{x}\sim P_d}[\omega f(D(\mathbf{x}))]+{\mathbb{E}}_{\tilde{\mathbf{x}}\sim P_g}[g(D(\tilde{\mathbf{x}}))],$$

$$L_{G}\geq L_{G}\,\big{\rvert}_{\,P_{g}=P_{d}}\,.$$

$$P_d(\mathbf{x})=\gamma \delta (\mathbf{x}-\mathbf{s})+(1-\gamma )\delta (\mathbf{x}-\mathbf{t}),$$

$$P_g(\mathbf{x})=(1-\gamma )\delta (\mathbf{x}-\mathbf{s})+\gamma \delta (\mathbf{x}-\mathbf{t}).$$

$$\displaystyle L_{G}\,\big{\rvert}_{\,P_{g}=P_{d}}$$

$$=\underset{D}{\max }{\int }_{\mathbf{x}}{P}_d(\mathbf{x})(f(D(\mathbf{x}))+g(D(\mathbf{x})))d\mathbf{x}$$

$$\displaystyle=\max_{D}\;\int_{\bm{x}}\big{(}\,(\gamma\,\delta(\bm{x}-\bm{s})+(1-\gamma)\,\delta(\bm{x}-\bm{t}))(f(D(\bm{x}))+g(D(\bm{x})))\,\big{)}\,d\bm{x}$$

$$\displaystyle=\max_{D}\;\big{(}\,\gamma\,(f(D(\bm{s}))+g(D(\bm{s})))+(1-\gamma)(f(D(\bm{t}))+g(D(\bm{t})))\,\big{)}$$

$$=\underset{y_1,y_2}{\max }\gamma (f(y_1)+g(y_1))+(1-\gamma )(f(y_2)+g(y_2))$$

$$=\underset{y_1}{\max }\gamma (f(y_1)+g(y_1))+\underset{y_2}{\max }(1-\gamma )(f(y_2)+g(y_2))$$

$$=\underset{y}{\max }f(y)+g(y)$$

$$=2\psi (\frac{1}{2}).$$

$$L_G$$

$$\displaystyle=\max_{D}\;\int_{\bm{x}}\big{(}\,(\gamma\,\delta(\bm{x}-\bm{s})+(1-\gamma)\,\delta(\bm{x}-\bm{t}))\,f(D(\bm{x}))+((1-\gamma)\,\delta(\bm{x}-\bm{s})+\gamma\,\delta(\bm{x}-\bm{t}))\,g(D(\bm{x}))\,\big{)}\,d\bm{x}$$

$$\displaystyle=\max_{D}\;\big{(}\,\gamma\,f(D(\bm{s}))+(1-\gamma)\,f(D(\bm{t}))+(1-\gamma)\,g(D(\bm{s}))+\gamma\,g(D(\bm{t}))\,\big{)}$$

$$\displaystyle=\max_{y_{1},y_{2}}\;\big{(}\,\gamma\,f(y_{1})+(1-\gamma)\,g(y_{1}))+(1-\gamma)\,f(y_{2})+\gamma\,g(y_{2})\,\big{)}$$

$$=\underset{y_1}{\max }\gamma f(y_1)+(1-\gamma )g(y_1))+\underset{y_2}{\max }(1-\gamma )f(y_2)+\gamma g(y_2)$$

Full text

1 Introduction

Generative adversarial networks (GANs) (Goodfellow et al., 2014) are a class of unsupervised machine learning algorithms. The core of a GAN is an adversarial loss that learns to measure the discrepancy between the real data distribution and the distribution of the generated samples. While adversarial losses are originally used to train a generative model (Goodfellow et al., 2014), their applications extend beyond unsupervised generative modeling to other supervised tasks such as conditional generative modeling (Mirza & Osindero, 2014) and discriminative modeling (dos Santos et al., 2017). Moreover, prior work has proposed a range of adversarial losses in different forms, motivated and derived from various existing divergences or distances. For example, the classic minimax GANs (Goodfellow et al., 2014) is motivated from the Jensen–Shannon divergence, Wasserstein GANs (Arjovsky et al., 2017) from the Wasserstein distance, least squares GANs (Mao et al., 2017) from the Pearson $\chi^{2}$ divergence, and $f$-GANs (Nowozin et al., 2016) from the $f$-divergence. Notably, these GAN formulations adopt different output activation functions in their training objectives. However, it remains unclear what types of output activation functions form a well-behaved adversarial loss. Moreover, apart from the output activation functions, the adversarial losses proposed in prior work differ also in the regularization approach adopted. Some representative examples include the gradient penalties (Gulrajani et al., 2017; Kodali et al., 2017; Mescheder et al., 2018) and the spectral normalization (Miyato et al., 2018). However, it remains unclear how the output activation functions and the regularization approach contribute respectively to the empirical performance of an adversarial loss.

In this paper, we aim to study the following two research questions:

Question 1**.**

What types of output activation functions form a well-behaved adversarial loss?

Question 2**.**

How different combinations of output activation functions and regularization approaches perform empirically against one another?

Our contributions to Question 1 are based on the variational divergence minimization perspective proposed by Nowozin et al. (2016) and the concept of adversarial divergence proposed by Liu et al. (2017). Specifically, we consider an adversarial loss well-behaved if it behaves as a divergence-like measure between the data and model distributions.111The model distribution is the distribution of the generated samples. It is implicitly defined by the generator (see Section 3.1). Using a generalized formulation proposed by Jolicoeur-Martineau (2019), we derive the necessary and sufficient conditions for an adversarial loss to be well-behaved. Our theoretical analysis reveals a large class of theoretically valid adversarial losses. Based on our findings, we propose two new adversarial losses—the absolute loss and the asymmetric loss. The absolute loss features a mixture of characteristics from two existing adversarial losses, while the asymmetric loss possesses distinct characteristics from other adversarial losses. While both losses do not originate from any underlying divergence, we will show in our empirical study that they can achieve similar performance against other adversarial losses.

For Question 2, some attempts have been made towards decoupling the effects of output activation functions and regularization approaches. For example, Fedus et al. (2018) showed that the original nonsaturating GAN (Goodfellow et al., 2014) can achieve comparable performance as the WGAN-GP model (Gulrajani et al., 2017) when using the same gradient penalty. However, due to the high computational cost for conducting a large-scale comparative study, Fedus et al. (2018); Lucic et al. (2018); Kurach et al. (2019) evaluate only five, seven and ten, respectively, combinations of output activation functions and regularization approaches. This necessitates the needs for an efficient way to compare adversarial losses. In view of the various tasks that adversarial losses are applied to, we propose to adopt the discriminative adversarial networks (DANs) for our comparative study as we can efficiently evaluate different adversarial losses using a standard evaluation metric such as the classification accuracy. For simplicity and computation cost concerns, we consider the MNIST handwritten digit classification problem (LeCun et al., 1998). With the proposed framework, we systematically evaluate 168 adversarial losses featuring the combinations of twelve output activation functions and fourteen regularization approaches. Moreover, we empirically study the effects of the Lipschitz constant (Arjovsky et al., 2017), penalty weights (Mescheder et al., 2018), momentum terms (Kingma & Ba, 2014). We also study the robustness of different regularization approaches to data imbalance.

Our findings suggest that there is no single winning combination of output activation functions and regularization approaches across all settings. The global lowest error rate is achieved by combining the relativistic hinge loss with the two-side local gradient penalty. Moreover, by comparing the minimum error rate achieved for each output activation function across different regularization approaches, we find that the hinge loss significantly outperforms the classic loss. Similarly, the coupled and local two-side gradient penalties significantly outperform other gradient penalties, no matter whether the spectral normalization is applied or not. Through comparing the maximum error rate achieved for each output activation function across different regularization approaches, we find that our proposed asymmetric loss is the most robust output activation function. Similarly, the spectral normalization improves the robustness for all gradient penalties. Although it is unclear whether the empirical results we obtain for DANs can generalize to unconditional or conditional GANs,222We note that even though the generator is no longer a generative model in a DAN, the discriminator is still trained to minimize the same loss function as that in a conditional GAN (see Section 3.7). we argue that the idea of adversarial losses is fairly generic and they can be used to train either generative or discriminative models. Hence, our empirical results still contribute to the goal towards a deeper understanding of adversarial losses. For reproducibility, all source code and hyperparameters can be found at https://github.com/salu133445/dan.

2 Related Work

Prior work has studied the working of adversarial losses from different perspectives. Some view adversarial losses from the perspective of variational divergence minimization (Nowozin et al., 2016). From this view, Nowozin et al. (2016) derived the $f$-GAN family based on the f-divergences. However, the output activation functions used for the data and model distribution need to be the conjugate function of each other in their formulations. In our theoretical analysis, we will relax this restriction and work on a more generalized formulation of adversarial losses. Further, Liu et al. (2017) proposed a broader concept of adversarial divergence and showed that if the objective function is an adversarial divergence with some conditions, then using a restricted discriminator family has a moment-matching effect. However, they did not show what output activation functions lead to a valid adversarial divergence. In this paper, we will derive the necessary and sufficient conditions for a well-behaved adversarial loss.

Some view adversarial losses from the perspective of integral probability metric view (Arjovsky & Bottou, 2017; Mroueh et al., 2017; Li et al., 2017; Mroueh & Sercu, 2017). In particular, Arjovsky & Bottou (2017) showed that by imposing a Lipschitz constraint to the discriminator, the training objective for the generator corresponds to minimizing the Wasserstein distance between the data and model distributions. In this formulation, the output activation functions used for the data and model distribution need to be the same. Later, some follow-up work proposed to enforce the Lipschitz constraint using gradient penalties (Gulrajani et al., 2017; Kodali et al., 2017; Wei et al., 2018; Mescheder et al., 2018) and spectral normalization (Miyato et al., 2018). Moreover, Fedus et al. (2018) suggested that these regularization approaches extend beyond the integral probability metric view and can be applied to other GAN formulations. Motivated by this observation, we will examine in our empirical study combining different output activation functions and regularization approaches to decouple their effects. Moreover, they suggested that the training dynamic of a GAN is closer to approaching a Nash equilibrium of the adversarial game than minimizing a specific divergence. Nonetheless, we can view an adversarial loss as a dynamic, divergence-like measure that quantifies the discrepancy between the data and model distributions. This view will serve as the foundation of our theoretical analysis.

Another relevant line of research are the comparative studies of adversarial losses. Fedus et al. (2018) showed that both the gradient penalty (Gulrajani et al., 2017) and spectral normalization (Miyato et al., 2018) can also improve the performance for a nonsaturating GAN (Goodfellow et al., 2014) in addition to the Wasserstein GAN (Arjovsky et al., 2017) they are originally proposed for. Later, Lucic et al. (2018) conducted a large-scale empirical study and showed that the seven GAN models considered in their study can reach similar scores given enough computation budgets. Similarly, Kurach et al. (2019) showed that the spectral normalization (Miyato et al., 2018) consistently improves the sample quality for the four output activation functions considered, while the gradient penalty (Gulrajani et al., 2017) require a large computation budget to find a good hyperparameter setting. However, due to the high computational cost for conducting a large-scale experiment, Fedus et al. (2018); Lucic et al. (2018); Kurach et al. (2019) examine only five, seven and ten, respectively, combinations of output activation functions and regularization approaches. In this paper, we will propose a simple comparative framework for efficiently comparing adversarial losses based on a discriminative adversarial network. With the proposed framework, we evaluate a comprehensive set of 168 combinations of twelve output activation functions and fourteen regularization approaches.

3 Background

In this section, we provide key background knowledge useful for our discussions in the rest of the paper.

3.1 Generative adversarial networks

A generative adversarial network (GAN) (Goodfellow et al., 2014) is a generative latent variable model that aims to learn a generative model $G:\mathcal{Z}\mapsto\mathcal{X}$ that maps some latent space $\mathcal{Z}$ to the data space $\mathcal{X}$. A discriminative model $D:\mathcal{X}\mapsto\mathbb{R}$ defined on $\mathcal{X}$ is trained alongside $G$ to provide guidance for $G$. Now, let $P_{d}$ denote the data distribution and $P_{z}$ the latent distribution. The original GAN is formulated as a two-player minimax game between the generator $G$ and discriminator $D$ as follows.

[TABLE]

Alternatively, let $P_{g}$ be the model distribution implicitly defined by $G(\bm{z})$ with $\bm{z}\sim P_{z}$.

[TABLE]

We will refer to this original formulation to the classic minimax GAN for the rest of this paper.

3.2 A variational divergence minimization view of adversarial losses

Some prior work investigated adversarial losses from the perspective of variational divergence minimization. Nowozin et al. (2016) proposed to minimize the following variational lower bound of a $f$-divergence derived by Nguyen et al. (2010):

[TABLE]

where $\mathcal{D}$ is an arbitrary set of functions $D:\mathcal{X}\mapsto\mathbb{R}$ and $f^{*}$ is the corresponding conjugate function for an $f$-divergence. Nowozin et al. (2016) showed that this can be formulated as an adversarial minimax game similar to a GAN:

[TABLE]

where $g_{f}:\mathbb{R}\mapsto dom(f^{*})$ is an output activation function specific to the $f$-divergence used. The $f$-GAN family given by Equation 4 includes some representative GAN formulations, including the classic minimax GANs (Goodfellow et al., 2014) and least-square GANs (Mao et al., 2017).

3.3 An integral probability metric view of adversarial losses

Some prior work has investigated viewing an adversarial loss as an estimator for an integral probability metric (IPM) (Müller, 1997). Arjovsky et al. (2017) considered the Wasserstein distance and proposed to minimize the following dual form given by the Kantorovich–Rubinstein duality:

[TABLE]

Arjovsky et al. (2017) then proposed the following Wasserstein GAN formulation:

[TABLE]

where $\mathcal{D}$ is the set of all 1-Lipschitz functions. Other examples of IPM-based GANs include McGANs (Mroueh et al., 2017), MMD GANs (Li et al., 2017) and Fisher GANs (Mroueh & Sercu, 2017).

3.4 Relativistic GANs

Jolicoeur-Martineau (2019) argued that the generator $G$ should also be trained to decrease the probability that the real samples are classified as real by the discriminator $D$. Based on this argument, they proposed two variants for GANs called the relativistic GANs and the relativistic average GANs. In this paper, we consider the relativistic average GAN as follows.

[TABLE]

where \hat{D}(\bm{x})=\sigma\big{(}D(\bm{x})-\mathbb{E}_{\tilde{\bm{x}}\sim P_{g}}\left[D(\tilde{\bm{x}})\right]\big{)}, \hat{D}(\tilde{\bm{x}})=\sigma\big{(}D(\tilde{\bm{x}})-\mathbb{E}_{\bm{x}\sim P_{d}}\left[D(\bm{x})\right]\big{)}, $\sigma(\cdot)$ is the sigmoid function. We also consider the relativistic average hinge GAN as follows.

[TABLE]

3.5 Gradient penalties

Originally proposed in (Gulrajani et al., 2017), a gradient penalty is a regularization approach to impose a soft Lipschitz constraint on the discriminator in a Wasserstein GAN (Arjovsky et al., 2017). Prior work has proposed different variants of gradient penalties (Kodali et al., 2017; Mescheder et al., 2018). In general, they take the following generalized form as a regularization term added to the discriminator loss function:

[TABLE]

where $\lambda\in\mathbb{R}$ is a hyperparameter, which we will refer to as the penalty weight, and $R:\mathbb{R}\mapsto\mathbb{R}$ is a real function. The distribution $p_{\hat{\bm{x}}}$ defines where the gradient penalty is enforced. This formulation covers some representative gradient penalties proposed in prior work, including the coupled gradient penalty used in the WGAN-GP model (Gulrajani et al., 2017), the local gradient penalty used in the DRAGAN model (Kodali et al., 2017), and the $R_{1}$ and $R_{2}$ gradient penalties (Mescheder et al., 2018), as summarized in Table 1. In addition, Figure 1 illustrates the support of $p_{\hat{\bm{x}}}$, i.e., where gradient penalty is imposed.

3.6 Spectral normalization

Another regularization approach for Wasserstein GANs (Arjovsky et al., 2017) is the spectral normalization proposed by Miyato et al. (2018). By normalizing the spectral norm of the weight matrix for each layer of the discriminator, the spectral normalization imposes a hard Lipschitz constraint on the discriminator. As compared to a gradient penalty, which imposes a local regularization as illustrated in Figure 1, the spectral normalization imposes a global regularization for the discriminator.

3.7 Discriminative adversarial networks

A discriminative adversarial network (dos Santos et al., 2017) is essentially a conditional GAN (Mirza & Osindero, 2014) where both the generator and discriminator are discriminative models. In a DAN, the generator aims to predict the label of a real data sample, whereas the discriminator takes as input either a real pair—“(real data, real label)” or a fake pair—“(real data, fake label)”, and aims to examine its authenticity. Mathematically, the generator in a DAN learns the mapping from the data space to the label space, i.e., $\mathcal{X}\mapsto\mathcal{Y}$, where $\mathcal{Y}$ denotes the label space. This is different from a conditional GAN, where the generator learns a mapping from the label space to the data space, i.e., $\mathcal{Y}\mapsto\mathcal{X}$. However, the discriminator in either a DAN or a conditional GAN learns the same mapping $\mathcal{X}\times\mathcal{Y}\mapsto\mathbb{R}$.

4 Theoretical Results

In this section, we present our theoretical results on Question 1: What types of output activation functions form a well-behaved adversarial loss? For the rest of the paper, we will adopt the following notations: $D$ denotes the discriminator and $G$ denotes the generator in a GAN. $P_{d}$ denotes the data distribution defined on the data space $\mathcal{X}$. $P_{z}$ denotes the latent distribution defined on the latent space $\mathcal{Z}$. $P_{g}$ denotes the model distribution implicitly defined by $G(\bm{z})$ with $\bm{z}\sim p_{z}$.

4.1 Generalized formulation of adversarial losses

Following Jolicoeur-Martineau (2019), we adopt a generalized form of adversarial losses as follows:

[TABLE]

where $f$, $g$ and $h$ are all real functions, i.e., $\mathbb{R}\mapsto\mathbb{R}$. We will refer to $f$, $g$ and $h$ as the output activation functions. This generalized formulation covers a wide range of adversarial losses proposed in prior work, including the classic minimax and nonsaturating GANs (Goodfellow et al., 2014), Wasserstein GANs (Arjovsky et al., 2017), least squares GANs (Mao et al., 2017) and hinge GANs (Lim & Ye, 2017; Tran et al., 2017), as summarized in Table 2.333Notably, this formulation does not cover the relativistic GANs proposed by Jolicoeur-Martineau (2019) (see Section 3.4), which we will include in our empirical comparative study in Section 6.

In the following theoretical analysis, we will focus only on the output activation functions $f$ and $g$. Specifically, we consider the minimax formulation where $h=g$:

[TABLE]

We note that from the perspective of either variational divergence minimization or integral probability metric, the generator should minimize the divergence-like measure estimated by the discriminator, and accordingly we have $h=g$. However, some prior work has investigated setting $h$ different from $g$. For example, the classic nonsaturating loss proposed by Goodfellow et al. (2014) has a different $h$ with opposite concavity as compared to the classic minimax loss (see Table 2). While our theoretical analysis concerns with only $f$ and $g$, we will empirically examine the effects of the output activation function $h$ in Section 6.2.

4.2 Desirable properties for adversarial losses

Following the perspective of variational divergence minimization, we can see that if the discriminator is able to reach optimality, the training objective for the generator is

[TABLE]

In principle, the discriminator in this formulation is responsible for providing a measure of the discrepancy between $P_{d}$ and $P_{g}$, which will then serve as the training objective for the generator to push $P_{g}$ towards $P_{d}$. Hence, we would like such an adversarial loss to be a divergence-like measure between $P_{g}$ and $P_{d}$. Following this view of adversarial divergence (Liu et al., 2017), we can define the following two properties of a well-behaved adversarial losses.

Property 1** (Weak desired property of a adversarial loss).**

For any data distribution $P_{d}$, the generator loss $L_{G}$ has a global minimum at $P_{g}=P_{d}$ with respect to the model distribution $P_{g}$.

Property 2** (Strong desired property of a adversarial loss).**

For any data distribution $P_{d}$, the generator loss $L_{G}$ has a unique global minimum at $P_{g}=P_{d}$ with respect to the model distribution $P_{g}$.

We can see that Property 2 makes $L_{G}-L^{*}_{G}$ a divergence between $P_{d}$ and $P_{g}$ for any fixed $P_{d}$, where L^{*}_{G}=L_{G}\,\big{\rvert}_{P_{g}=P_{d}} is a constant term irrelevant to optimization. In contrast, Property 1 provides a weaker version when the uniqueness of the global minimum is not guaranteed.

4.3 The $\Psi$ and $\psi$ functions

In order to derive the necessary and sufficient conditions for Property 1 and Property 2, we first observe from Equation 15 that

[TABLE]

Assuming $D$ can be any function that maps from $\mathcal{X}$ to $\mathbb{R}$,444This assumption does not hold if $D$ is parameterized as a neural network. See the discussions in Section 4.7. we can let $y=D(\bm{x})$ be a variable independent of $\bm{x}$. Then, we can exchange the order of the integral and maximum in Equation 16 and obtain

[TABLE]

Next, rearranging Equation 17 we obtain

[TABLE]

Now, let $y=D(\bm{x})$ and $\gamma=\frac{P_{d}(\bm{x})}{P_{d}(\bm{x})+P_{g}(\bm{x})}$ (note that $\gamma=\frac{1}{2}$ if and only if $P_{d}(\bm{x})=P_{g}(\bm{x})$). We then obatin

[TABLE]

Let us consider the latter term inside the integral and define the following two functions:

[TABLE]

where $\gamma\in[0,1]$ and $y\in\mathbb{R}$. We visualize in Figure 2 the $\Psi$ functions and Figure 3 the $\psi$ functions for the adversarial losses considered in this paper.

4.4 Necessary and sufficient conditions for a well-behaved adversarial loss

With the $\psi$ function defined by Equation 21, we can now derive the necessary and sufficient conditions for a well-behaved adversarial loss. All proofs can be found in Appendix A. First, Theorems 1 and 2 provide the necessary conditions for a well-behaved adversarial loss.

Theorem 1** (Necessary conditions of the weak desired property of an adversarial loss).**

If Property 1 holds, then for any $\gamma\in[0,1]$, $\psi(\gamma)+\psi(1-\gamma)\geq 2\,\psi(\frac{1}{2})$.

Theorem 2** (Necessary conditions of the strong desired property of an adversarial loss).**

If Property 2 holds, then for any $\gamma\in[0,\frac{1}{2})\cup(\frac{1}{2},1]$, $\psi(\gamma)+\psi(1-\gamma)>2\,\psi(\frac{1}{2})$.

For the sufficient conditions, we have Theorems 3 and 4 as follows.

Theorem 3** (Sufficient conditions of the weak desired property of an adversarial loss).**

If $\psi(\gamma)$ has a global minimum at $\gamma=\frac{1}{2}$, then Property 1 holds.

Theorem 4** (Sufficient conditions of the strong desired property of an adversarial loss).**

If $\psi(\gamma)$ has a unique global minimum at $\gamma=\frac{1}{2}$, then Property 2 holds.

With Theorems 3 and 4, we can check if a pair of output activation functions $f$ and $g$ form a well-behaved adversarial loss by finding their corresponding $\psi$ function.555Note that Theorem 4 still holds for the Wasserstein loss even if its $\psi$ function is only defined at $\gamma=0.5$. Further, we have Theorem 5 for a more specific guideline for choosing or designing the output activation functions $f$ and $g$.

Theorem 5** (Alternative sufficient conditions of the strong desired property of an adversarial loss).**

If (1) $f^{\prime\prime}+g^{\prime\prime}\leq 0$ and (2) there exists some $y^{*}\in\mathbb{R}$ such that $f(y^{*})=g(y^{*})$ and $f^{\prime}(y^{*})=-g^{\prime}(y^{*})\neq 0$, then Property 2 holds.

In other word, a pair of output activation functions $f$ and $g$ form a well-behaved adversarial loss as long as (1) the sum of $f$ and $g$ is concave downward and (2) we can find a point where they take the same value while having exact opposite nonzero derivatives. Intuitively, the second part of the prerequisites reflects the adversarial nature of the game between the discriminator and generator as it requires their objectives are opposite at some $y^{*}\in\mathbb{R}$.

Theorem 5 reveals a large class of theoretically valid adversarial losses. While such a theoretical analysis has not been done on the generalized form of adversarial losses in Equations 12 and 13, it happens that all the adversarial losses listed in Table 2 have such desirable properties, and we show in Table 2 their corresponding $y^{*}$ in Theorem 5. Moreover, we note that Nowozin et al. (2016) also derived a large class of adversarial losses based on the $f$-divergence. However, in their formulations, $f$ and $g$ need to be the conjugate function of each other. We discuss the connections to $f$-GANs in Appendix B Similarly, IPM-based GAN (Arjovsky et al., 2017) require $f$ and $g$ to be the same. In contrast, the prerequisites of Theorem 5 is less restricted. For example, the hinge loss (Lim & Ye, 2017; Tran et al., 2017) does not fall into either the $f$-GAN or IPM-based GAN classes, while it satisfies the prerequisites of Theorem 5 and is thus a well-behaved adversarial loss based on our theoretical results.

4.5 Analyzing the adversarial game by the $\Psi$ functions

Apart from deriving the necessary and sufficient conditions for a well-behaved adversarial loss, the $\Psi$ and $\psi$ functions defined in Equations 21 and 20 also provide some new insights into the adversarial behaviors of the generator and discriminator. If we follow Equation 19 and consider $y=D(\bm{x})$ and $\gamma(\bm{x})=\frac{P_{d}(\bm{x})}{P_{d}(\bm{x})+P_{g}(\bm{x})}$, then the discriminator can be viewed as maximizing $\Psi$ along the $y$-axis in Figure 2. On the other hand, since the generator is trained to push $P_{g}$ towards $P_{d}$, it can be viewed as minimizing $\Psi$ along the $\gamma$-axis. Following this view, the saddle-shaped landscape of the $\Psi$ functions presented in Figure 2 reflects the minimax nature of the adversarial game. Moreover, they all have saddle points at $\gamma=\frac{1}{2}$, which happens when $P_{d}(\bm{x})=P_{g}(\bm{x})$. In the ideal case when the discriminator can be trained till optimality in each iteration, we will always stay on the domain of $\psi$, i.e., the green line. In this case, the generator can be viewed as minimizing $\Psi$ along the green line.666Since $L_{G}$ is an integral over all possible $\bm{x}\in\mathcal{X}$, this adversarial game is played in a high dimensional space. Finally, the neighborhoods around the saddle points of $\Psi$ for different adversarial losses possess distinct characteristics, and such difference may affect the game dynamics and optimization stability. Further investigation on the properties of $\Psi$ and $\psi$ may lead insights into why and how different output activation functions perform differently in practice, which we will leave as future work.

4.6 Designing adversarial losses by the $\Psi$-landscape

From the analysis presented in the previous sections, we see that the $\Psi$ function reflects some important characteristics of an adversarial loss. By observing and designing the landscape of the $\Psi$ function, we propose the following two new adversarial losses that comply with Theorem 5.

• •

First, the absolute loss is obtained by replacing the square functions in the least squares loss with absolute functions. Specifically, we have $f(y)=-h(y)=-|1-y|$, $g(y)=-|y|$. As shown in Figure 2(e), its $\Psi$-landscape features a bounded positive-valued region (i.e., the red-colored region) of similar shape to that of the least squares loss. It also has a bounded global minima region (i.e., the yellow dashed lines) similar to that of the hinge loss. We propose this loss as its output activation functions are simple, piecewise linear.

• •

Second, the asymmetric loss is obtained by negating the output activation function $f$ of a Wasserstein for $y\in\mathbb{R}^{+}$. Specifically, we have $f(y)=-|y|$, $g(y)=h(y)=-y$. As can be seen from Figure 2(f), it features an asymmetric, non-saddle shaped $\Psi$-landscape distinct from other adversarial losses. Its $\Psi$-landscape is similar to that of the Wasserstein loss, but the positive part of $y$ is all negative-valued. As all adversarial losses in Table 2 has symmetric $f$ and $g$, we propose this loss function to show that the output activation functions $f$ and $g$ for a well-behaved adversarial loss does not necessarily need to be symmetric.

We will include these two new losses in our empirical study in Section 6.2. While these two losses do not originate from any underlying divergence, we will show in Section 6.2 that they can achieve similar empirical performance against other adversarial losses, and that an adversarial loss does not necessarily need to have a saddle-shaped $\Psi$-landscape.

4.7 Limitations of the theoretical results

We want to discuss some of the limitations of our theoretical analysis presented in the previous sections. First, while our analysis originates from a variational divergence minimization perspective, the GAN training dynamic in practice has been shown to be closer to approaching a Nash equilibrium of the adversarial game than minimizing a specific divergence, as suggested by Fedus et al. (2018). While Theorem 5 reveals a large class of theoretically valid adversarial losses, their optimization and convergence properties remain unclear and require further investigations. Second, our derivation in Section 4.3 is based on the assumption that the discriminator $D$ can be any function. However, this does not hold in general. In practice, the discriminator is usually parameterized as a neural network, thus having limited expressiveness. Moreover, the regularization approaches adopted may further constraint the set of functions that $D$ can possibly be. For example, IPM-based GANs consider only 1-Lipschitz functions for $D$ (Arjovsky et al., 2017). Finally, we note that the sufficient conditions provided in Theorem 5 are not tight. To examine the tightness of these sufficient conditions, we will empirically examine in Section 6.6 some cases when the prerequisites of Theorem 5 do not hold.

5 A Comparative Framework for Adversarial Losses

In Section 4, we derive the necessary and sufficient conditions for a well-behaved adversarial loss. While the analysis reveals a large class of theoretical valid adversarial losses, it brings us to the next question—how does these different adversarial losses perform against one another? Although we focus only on the output activation functions in our theoretical analysis, a pair of adversarial losses can differ not only in their output activation functions but also the regularization approaches adopted. While Fedus et al. (2018); Lucic et al. (2018); Kurach et al. (2019) has attempted to decouple their effects, they evaluate only five, seven and ten, respectively, combinations of output activation functions and regularization approaches in their experiments due to the high computational cost for conducting the comparative study. This necessitates the needs for an efficient way to empirically compare different adversarial losses. Motivated by the various domains that the adversarial losses have been applied to, we propose to compare adversarial losses through using discriminative adversarial networks (DANs).

In this paper, we will work on the MNIST handwritten digit classification problem (LeCun et al., 1998). As illustrated in Figure 4, the generator $G$ in a DAN takes a sample $\bm{x}\in\mathcal{X}$ as input and predict a fake label $c\in C$, where $C$ is the set of all possible classes. The discriminator takes a pair $(\bm{x},c)$ ($\bm{x}\in\mathcal{X}$, $c\in C$) as input, where the label can either be the real label or a label predicted by the generator, and aims to examine if it is a real pair. Mathematically, the discriminator of a DAN learns the same mapping $\mathcal{X}\times\mathcal{Y}\mapsto\mathbb{R}$ as that of a conditional GAN does. The generator, a classification model in this example, is then trained by the supervision signal provided by the discriminator.

Although we work on a classification task in this paper, the proposed framework is generic to the underlying task as long as the evaluation metrics for the task are well defined. Moreover, adopting a DAN in this framework offers two additional advantages over a GAN. First, it avoids the high computation cost for training a GAN as the generator in a DAN is a discriminative model rather than a generative model in a GAN. Second, we can use standard evaluation metrics to evaluate a DAN. Hence, its performance can be easier evaluated than a GAN, which is usually assessed via more complex metrics such as the Inception Score (Salimans et al., 2016) and Fréchet Inception Distance (Heusel et al., 2017).

6 Empirical Results

Based on the comparative framework proposed in Section 5, we present in this section our empirical results on Question 2: How different combinations of output activation functions and regularization approaches perform empirically against one another? In addition, we use the proposed framework to examine the effects of penalty weights, Lipschitz constants and momentum terms in the optimizer. Moreover, we examine the tightness of the sufficient conditions provided by Theorem 5 and the robustness of different regularization approaches to data imbalance.

6.1 Implementation details

We implement the comparative framework described in Section 5 on the MNIST handwritten digit classification problem (LeCun et al., 1998). We implement both the generator $G$ and discriminator $D$ as convolutional neural networks (CNNs). In addition, we use the same network architectures across all experiments, as summarized in Table 3. Moreover, we concatenate the label vector to the input of each layer in the discriminator. To speed up and stabilize the training, we use the batch normalization (Ioffe & Szegedy, 2015) in the generator, and the layer normalization (Ba et al., 2016) in the discriminator, which is disabled when the spectral normalization (Miyato et al., 2018) is used. For all the gradient penalties, we use Euclidean norms and a penalty weight $\lambda=10$ in Equation 11. In addition, we use a Lipschitz constant $k=1$ for the coupled and local gradient penalties (Gulrajani et al., 2017; Kodali et al., 2017). We set the hyperparameter $c$ for the local gradient penalties (Kodali et al., 2017) to $0.01$ (see Table 1). We use ReLUs (Glorot et al., 2011) as the activation functions for all layers except the last layer of the generator, which uses a softmax function, and the last layer of the discriminator, which has no activation function.777We do not apply any activation function to the output layer of the discriminator because the activation function is instead implemented in the loss terms in our formulation (see Equations 13 and 12).. For the training, we use the Adam optimizer (Kingma & Ba, 2014) with $\alpha=0.001$, $\beta_{1}=0$ and $\beta_{2}=0.9$. We implement all the models in Python and TensorFlow (Abadi et al., 2016). All the experiments are run on NVIDIA 1080 Tis with a batch size of $64$. We alternatively update $G$ and $D$ once in each iteration and train the model for 100,000 generator steps, which takes roughly two hours on an NVIDIA 2080 Ti. Thanks to the speed, we repeat each experiment for ten runs to reduce the effects of randomness on our experimental results. For evaluation, we report the mean error rate and its 95% confidence interval. For reproducibility, all source code and hyperparameters can be found at https://github.com/salu133445/dan.

6.2 Decoupling the effects of output activation functions and regularization approaches

In the first experiment, we aim to study how different output activation functions and regularization approaches perform against one another. Moreover, in order to decouple their effects, we evaluate a comprehensive set of 168 combinations of twelve output activation functions and fourteen regularization approaches. For the output activation functions, we consider the classic minimax and nonsaturating losses (Goodfellow et al., 2014), Wasserstein loss (Arjovsky et al., 2017),888In this paper, we will refer to the Wasserstein loss as a type of output activation functions defined in Table 2. The Lipschitz constraints introduced by Arjovsky et al. (2017) are decoupled and imposed by different regularization approaches. least squares loss (Mao et al., 2017), hinge loss (Lim & Ye, 2017; Tran et al., 2017), relativistic average and relativistic average hinge losses (Jolicoeur-Martineau, 2019), as well as the absolute and asymmetric losses we propose in Section 4.5. Moreover, while our theoretical analysis focuses only on the output activation functions $f$ and $g$, we also aim to examine the effects of the output activation function $h$ for the generator. Specifically, we examine pairing the classic and hinge losses with the following three variants of output activation function $h$: minimax—$h(y)=g(y)$, nonsaturating—$h(y)=\log(1+e^{-y})$, and linear—$h(y)=-y$. For the regularization approaches, we consider the coupled gradient penalties (Gulrajani et al., 2017), local gradient penalties (Kodali et al., 2017), R1 and R2 gradient penalties (Mescheder et al., 2018) and the spectral normalization (Miyato et al., 2018). For the coupled and local gradient penalties, we examine both the two-side and one-side versions (see Table 1). In addition, we also examine the combinations of the spectral normalization with different gradient penalties.

We report in Table 4 the results of the comparative study.999As a baseline, we train the generator $G$ with the cross entropy loss, and the minimum error rate it achieves is 4.12%. Our empirical results suggest that there is no single winning combination of output activation functions and regularization approach across all settings. The global lowest error rate is achieved by the combination of the relativistic hinge loss and two-side local gradient penalty. However, either relativisitic hinge loss or two-side local gradient penalty does not always achieve the lowest error rate when combined with other output activation functions or regularization approaches. Moreover, some combinations result in a notable high error rates, e.g., “classic minimax loss + R1 gradient penalty,” “least squares loss + R1 gradient penalty,” and “least squares loss + R2 gradient penalty.”

With respect to the output activation functions, we find that the classic minimax and nonsaturating losses never achieve the lowest error rates with any regularization approach. In general, The hinge minimax and relativistic average hinge losses achieve lower error rates than other losses, achieving the lowest error rates for three and four regularization approaches, respectively. The relativistic average loss outperforms both the classic minimax and nonsaturating losses across all regularization approaches, while the relativistic average hinge loss does not always outperform the standard hinge losses. The proposed absolute and asymmetric losses are more robust to different regularization approaches, representing the only two losses that achieve a mean error rate under 12% with any regularization approach. Regarding the output activation function $h$ for the generator, the three variants perform similarly and the differences do not reach statistical significance for most regularization approaches except the unregularized case. For the unregaularized case, the minimax version significantly outperforms the other two variants.

With respect to the regularization approaches, we find that the coupled and local gradient penalties outperform the R1 and R2 gradient penalties across nearly all output activation functions, no matter whether the spectral normalization is used or not. The two-side local penalty achieves the lowest error rate for six output activation functions. Combining either the coupled or local gradient penalty with the spectral normalization usually leads to higher error rates than using the gradient penalty only. The one-side gradient penalties lead to higher error rates than their two-side counterparts for all output activation functions.

To better analyze the empirical results, we also report in Figure 5(a) the minimum mean error rates achieved for each output activation function across different regularization approaches. Similarly, Figure 5(b) the minimum mean error rates achieved for regularization approach across different output activation functions. First, the hinge loss significantly outperforms the classic loss, no matter what the generator loss functions are. Second, the least squares loss performs significantly worse than the other output activation functions, while the hinge, Wasserstein, relativistic and absolute losses perform similarly. Third, the two two-sided gradient penalties significantly outperform other gradient penalties, no matter whether the spectral normalization is applied or not. Finally, combining the spectral normalization with gradient penalties does not necessarily lead to lower error rates.

Similar to Figure 5, we also report in Figure 6 the maximum mean error rates achieved. Note that we exclude the unregularized versions in Figure 6 to provide more meaningful analysis as the unregularized versions tends to fail easily. This serves as an indicator to the robustness and stability of each output activation function and regularization approach when combined with different components. First, the least squares loss are not robust to regularization approaches.101010Similarly, Lucic et al. (2018) also found that the least-squared GAN is not robust to random weight initialization. Second, while the relativistic hinge loss achieves the global lowest error rates when combined with the two-way local gradient penalties, it is not as robust to the asymmetric loss, which achieves the second lowest error rates when combined with the two-way coupled gradient penalties. Third, the spectral normalization improves the robustness for all gradient penalties.

Further, to investigate how the training dynamic is affected by the regularization approach, we present in Figure 7 the training progress for the nonsaturating loss with different regularization approaches. We can see that the coupled and local gradient penalties result in a stable training dynamic, no matter whether the spectral normalization is used or not. The R1 and R2 gradient penalties are too strong and halt the training early. This is possibly because they encourage the discriminator to have small gradients, and thus the gradients for both the generator and discriminator may vanish when the data and model distributions are close enough. In addition, combining either the R1 or R2 gradient penalty with the spectral normalization leads to an unstable training dynamic.

6.3 Effects of the Lipschitz constants

In this experiment, we examine the effects of the Lipschitz constant $k$ in the coupled and local gradient penalties (Gulrajani et al., 2017; Kodali et al., 2017). We report in Figure 8 the results for $k=0.01$, $0.1$, $1$, $10$, $100$ using the classic nonsaturating loss. We can see that the error rate increases as $k$ goes away from $1.0$, suggesting that $k=1$ is a good default value. In addition, the two-side gradient penalties achieve lower error rates when $k$ is set properly, while they are more sensitive to $k$ than their one-side counterparts.

6.4 Effects of the penalty weights

We examine in this experiment the effects of the penalty weights $\lambda$ for the R1 and R2 gradient penalties (Mescheder et al., 2018). We show in Figure 9 the results for $\lambda=0.01,0.1,1,10,100$ using the classic nonsaturating, Wasserstein and hinge losses. We can see that the R2 gradient penalty generally outperform the R1 gradient penalty. However, they are both sensitive to the value of $\lambda$. Moreover, when the spectral normalization is used, the error rate increases as $\lambda$ increases, which implies that the R1 and R2 gradient penalties do not work well with the spectral normalization.

6.5 Effects of the momentum terms of the optimizers

We observe a trend in prior work towards using a smaller momentum (Radford et al., 2016), no momentum (Arjovsky et al., 2017; Gulrajani et al., 2017; Miyato et al., 2018; Brock et al., 2019) or a negative momentum (Gidel et al., 2019) in GAN training. Hence, we want to examine in this experiment the effects of the momentum term $\beta_{1}$ in an Adam optimizer (Kingma & Ba, 2014) with our proposed framework. Further, since the generator and discriminator are trained by distinct objectives, we also investigate setting different momentum values for them. We present in Figure 10 the results for all combinations of $\beta_{1}=-0.5,0.0,0.5,0.9$ for the generator and discriminator, using the classic nonsaturating loss along with the spectral normalization and coupled gradient penalties for regularization. We can see that for the two-side coupled gradient penalty, using a larger momentum in both the generator and discriminator leads to lower error rates. However, there is no specific trend for the one-side coupled gradient penalty.

6.6 Examining the necessary conditions for desirable adversarial loss functions

As discussed in Section 4.4, the sufficient conditions provided by Theorem 5 are not tight. That is, there are other valid output activation functions that satisfies Property 2 but not the conditions in Theorem 5. In order to examine the tightness of these sufficient conditions in Theorem 5, we examine in this experiment the cases when the prerequisites in Theorem 5 are violated. Specifically, we change the training objective for the discriminator into the following $\omega$-weighted loss:

[TABLE]

where $\omega\in\mathbb{R}^{+}\setminus\{1\}$ is a constant. Note that the training objective for the generator remains the same. We can see that the prerequisites in Theorem 5 do not hold when $\omega\neq 1$.111111See Appendix C for the graphs of the corresponding $\Psi$ and $\psi$ functions for $\omega=0.5,0.9,1.0,1.1,2.0$. We consider the classic nonsaturating, the Wasserstein and hinge losses, using the spectral normalization for regularization. We show in Table 5 the results for $\omega=0.5,0.9,1.0,1.1,2.0$. We can see that the original losses result in the lowest error rates as compared to the $\omega$-weighted loss. While the Wasserstein loss fails with error rates over 50% when $\omega\neq 1$, the $\omega$-weighted versions of the nonsaturing and hinge losses achieve a higher yet still acceptable error rate. This suggests that even if the $\omega$-weighted loss violates the conditions in Theorem 5, it can still provide some supervision signal for training. Moreover, the error rates generally increase as $\omega$ goes away from $1.0$. These results suggest that although Theorem 5 reveals a large class of theoretical valid adversarial losses, the sufficient conditions it provide are not tight.

6.7 Limitations of the empirical results

While the proposed comparative framework allows us to efficiently compare adversarial losses without high computation costs, one of the main limitations of our empirical results is that it is unclear whether the empirical results we obtain for DANs can be generalized to conditional and unconditional GANs. Although the discriminators in a DAN and a conditional GAN are trained to optimize the same objective, learning the same mapping $\mathcal{X}\times\mathcal{Y}\mapsto\mathbb{R}$, the generators are trained to learn the opposite mappings—the generator in a DAN learns the mapping $\mathcal{X}\mapsto\mathcal{Y}$, while the generator in a conditional GAN learns the mapping $\mathcal{Y}\mapsto\mathcal{X}$. Nonetheless, we argue that the idea of adversarial losses is rather generic and can be applied to various tasks, either generative or discriminative. Our empirical results still contribute to the goal towards a deeper understanding of adversarial losses.

Another limitation of our empirical study is the lack of hyperparameter search due to computation budget. In our experiments, we follow the recommended hyperparameter settings for each regularization approach. However, as suggested by Lucic et al. (2018) and Kurach et al. (2019), most GAN models can reach similar scores with enough hyperparameter optimization and random restarts. Accordingly, certain models might be able to reach a lower error rate with proper hyperparameter optimizations. Nevertheless, we argue that this phenomenon is less presented in our case as we use the adversarial losses to train the generator to perform a much simpler task of handwritten digit classification. In general, we observe a small variance on most of our empirical results except certain combinations.

7 Conclusion

In the first part of this paper, following the perspective of variational divergence minimization, we derived the necessary and sufficient conditions of a well-behaved adversarial loss in a generalized formulation. Our theoretical analysis reveals a large class of adversarial losses that includes a range of adversarial losses proposed in prior work. In the second part of this paper, we proposed a simple comparative framework for adversarial losses based on discriminative adversarial networks. With the proposed framework, we evaluated a comprehensive set of 168 adversarial losses featuring the combinations of twelve output activation functions and fourteen regularization approaches. Our empirical findings suggest that there is no single winning combination of output activation functions and regularization approaches across all settings. The theoretical and empirical results presented in this paper may together serve as a reference for choosing or designing adversarial losses in future research.

Appendix A Proofs of Theorems

Theorem 1** (Necessary conditions of the weak desired property of an adversarial loss).**

If Property 1 holds, then for any $\gamma\in[0,1]$, $\psi(\gamma)+\psi(1-\gamma)\geq 2\,\psi(\frac{1}{2})$.

Proof.

Since Property 1 holds, we have for any fixed $P_{d}$,

[TABLE]

Let us consider

[TABLE]

for some $\gamma\in[0,1]$ and $\bm{s},\bm{t}\in\mathcal{X},\bm{s}\neq\bm{t}$. Then, we have

[TABLE]

Moreover, we have

[TABLE]

(Note that we can obtain Equation 30 from Equation 29 and Equation 37 from Equation 36 because $D$ can be any function and thus $D(\bm{s})$ is independent of $D(\bm{t})$.)

As Equation 23 holds for any fixed $P_{d}$, by substituting Equation 33 and Equation 38 into Equation 23, we get

[TABLE]

for any $\gamma\in[0,1]$, which concludes the proof. ∎

Theorem 2** (Necessary conditions of the strong desired property of an adversarial loss).**

If Property 2 holds, then for any $\gamma\in[0,\frac{1}{2})\cup(\frac{1}{2},1]$, $\psi(\gamma)+\psi(1-\gamma)>2\,\psi(\frac{1}{2})$.

Proof.

Since Property 2 holds, we have for any fixed $P_{d}$,

[TABLE]

Following the proof of Theorem 1, consider

[TABLE]

for some $\gamma\in[0,1]$ and some $\bm{s},\bm{t}\in\mathcal{X},\bm{s}\neq\bm{t}$. It can be easily shown that $P_{g}=P_{d}$ if and only if $\gamma=\tfrac{1}{2}$.

As Equation 41 holds for any fixed $P_{d}$, by substituting Equation 42 and Equation 43 into Equation 41, we get

[TABLE]

for any $\gamma\in[0,\frac{1}{2})\cup(\frac{1}{2},1]$, concluding the proof. ∎

Theorem 3** (Sufficient conditions of the weak desired property of an adversarial loss).**

If $\psi(\gamma)$ has a global minimum at $\gamma=\frac{1}{2}$, then Property 1 holds.

Proof.

First, we see that

[TABLE]

On the other had, by definition we have

[TABLE]

Since $D$ can be any function that maps from $\mathcal{X}$ to $\mathbb{R}$,121212Note that this is not always true if some constraints are imposed on $D$. we can let $y=D(\bm{x})$ be a variable independent of $\bm{x}$. Thus, we have

[TABLE]

Since $\frac{P_{d}(\bm{x})}{P_{d}(\bm{x})+P_{g}(\bm{x})}\in[0,1]$, we have

[TABLE]

As $\psi(\gamma)$ has a global minimum at $\gamma=\frac{1}{2}$, now we have

[TABLE]

Finally, combining Equation 50 and Equation 58 yields

[TABLE]

which holds for any $P_{d}$, thus concluding the proof. ∎

Theorem 4** (Sufficient conditions of the strong desired property of an adversarial loss).**

If $\psi(\gamma)$ has a unique global minimum at $\gamma=\frac{1}{2}$, then Property 2 holds.

Proof.

Since $\psi(\gamma)$ has a unique global minimum at $\gamma=\frac{1}{2}$, we have for any $\gamma\in[0,\frac{1}{2})\cup(\frac{1}{2},1]$,

[TABLE]

When $P_{g}\neq P_{d}$, there must be some $\bm{x}_{0}\in\mathcal{X}$ such that $P_{g}(\bm{x}_{0})\neq P_{d}(\bm{x}_{0})$. Thus, $\frac{P_{d}(\bm{x}_{0})}{P_{d}(\bm{x}_{0})+P_{g}(\bm{x}_{0})}\neq\frac{1}{2}$, and thereby $\psi\left(\frac{P_{d}(\bm{x}_{0})}{P_{d}(\bm{x}_{0})+P_{g}(\bm{x}_{0})}\right)>\psi(\frac{1}{2})$. Now, by Equation 55 we have

[TABLE]

Finally, combining Equation 50 and Equation 64 yields

[TABLE]

which holds for any $P_{d}$, thus concluding the proof. ∎∎

Theorem 5** (Alternative sufficient conditions of the strong desired property of an adversarial loss).**

If $f^{\prime\prime}+g^{\prime\prime}\leq 0$ and there exists some $y^{*}$ such that $f(y^{*})=g(y^{*})$ and $f^{\prime}(y^{*})=-g^{\prime}(y^{*})\neq 0$, then Property 2 holds.

Proof.

First, we have by definition

[TABLE]

By taking the partial derivatives, we get

[TABLE]

We know that there exists some $y^{*}$ such that

[TABLE]

1. (i)

By Equation 67 and Equation 68, we see that

[TABLE]

Now, by Equation 72 we know that $\Psi$ is constant when $y=y^{*}$. That is, for any $\gamma\in[0,1]$,

[TABLE] 2. (ii)

Because $f^{\prime\prime}+g^{\prime\prime}\leq 0$, by Equation 69 we have

[TABLE]

By Equation 73 and Equation 75, we see that $y^{*}$ is a global minimum point of \Psi\big{\rvert}_{\gamma=\tfrac{1}{2}}. Thus, we now have

[TABLE] 3. (iii)

By Equation 68, we see that

[TABLE]

Since $f^{\prime}(y^{*})\neq 0$, we have

[TABLE]

This shows that for any $\gamma\in[0,\frac{1}{2})\cup(\frac{1}{2},1]$, there must exists some $y^{\circ}$ such that

[TABLE]

And by definition we have

[TABLE]

Hence, by Equation 83 and Equation 84 we get

[TABLE]

Now, combining Equation 73, Equation 78 and Equation 86 yields

[TABLE]

Finally, by Theorem 4, we know that Property 2 holds, which concludes the proof. ∎

Appendix B Connections to $f$-divergence

Nowozin et al. (2016) proposed the $f$-GAN family based on the $f$-divergence, which can be formulated as

[TABLE]

where $\hat{f}$ is convex and lower-semicontinuous and $\hat{f}(1)=0$.131313Here, we adopt the first formulation proposed by Nowozin et al. (2016), where the output activation $g_{f}$ is not included. Following the definition given by Equation 12, we have $f(y)=y$ and $g(y)=-\hat{f}^{*}(y)$. We show some representative $f$-GANs in Table 6 and their corresponding output activation functions.

To connect $f$-GANs to our theoretical results, we first derive their corresponding $\Psi$ and $\psi$ functions:

[TABLE]

(Note that $\hat{f}^{*}(t)=\sup_{x\in dom_{f}}\{xt-f(x)\}$ by definition and $\hat{f}^{**}=\hat{f}$. Also, note that $\gamma\in[0,1]$.)

We then examine whether these $f$-GANs fall into out theoretical framework. From Tables 6 and 7, we can see that many $f$-GAN instances satisfies the conditions in Theorems 4 and 5 and hence are considered well-behaved adversarial losses under our theoretical analysis. However, not all $f$-GANs satisfies Theorems 4 and 5. For example, the corresponding $\Psi$ functions for Kullback–Leibler (KL) and reverse KL divergences do not have a global minimum at $\gamma=\frac{1}{2}$.

Appendix C Graphs of the $\Psi$ and $\psi$ Functions for the $\omega$-weighted Losses

Figures 11 and 12 show the graphs of $\Psi$ and $\psi$, respectively, for the $\omega$-weighted versions of the classic, Wasserstein and hinge losses.