Generative Adversarial Minority Oversampling

TL;DR

This paper introduces a novel adversarial oversampling method for deep learning that generates synthetic minority class samples to improve classification performance on imbalanced datasets.

Contribution

It proposes a three-player adversarial framework involving a convex generator, classifier, and discriminator to generate effective minority class samples in deep learning.

Findings

Improves minority class classification accuracy

Generates samples near class boundaries

Effective on multiple imbalanced image datasets

Abstract

Class imbalance is a long-standing problem relevant to a number of real-world applications of deep learning. Oversampling techniques, which are effective for handling class imbalance in classical learning systems, can not be directly applied to end-to-end deep learning systems. We propose a three-player adversarial game between a convex generator, a multi-class classifier network, and a real/fake discriminator to perform oversampling in deep learning systems. The convex generator generates new samples from the minority classes as convex combinations of existing instances, aiming to fool both the discriminator as well as the classifier into misclassifying the generated samples. Consequently, the artificial samples are generated at critical locations near the peripheries of the classes. This, in turn, adjusts the classifier induced boundaries in a way which is more likely to reduce misclassification from the minority classes. Extensive experiments on multiple class imbalanced image datasets establish the efficacy of our proposal.

Tables

Table 1. Table 1: Comparison of classification performance of CE and LS variants of classifiers on MNIST and Fashion-MNIST datasets.

	MNIST				Fashion-MNIST
Algorithm	CE		LS		CE		LS
	ACSA	GM	ACSA	GM	ACSA	GM	ACSA	GM
Baseline CN	0.88$\pm$0.01	0.87$\pm$0.02	0.88$\pm$0.01	0.86$\pm$0.01	0.82$\pm$0.01	0.80$\pm$0.01	0.81$\pm$0.01	0.79$\pm$0.01
SMOTE+CN	0.88$\pm$0.02	0.87$\pm$0.03	0.89$\pm$0.01	0.89$\pm$0.01	-	-	-	-
Augment+CN	-	-	-	-	0.82$\pm$0.01	0.78$\pm$0.01	0.82$\pm$0.01	0.78$\pm$0.01
DOS	-	-	-	-	0.82$\pm$0.01	0.79$\pm$0.01	0.81$\pm$0.01	0.79$\pm$0.02
(cGAN/cDCGAN)+CN	0.88$\pm$0.01	0.87$\pm$0.01	0.89$\pm$0.01	0.88$\pm$0.01	0.81$\pm$0.02	0.78$\pm$0.01	0.82$\pm$0.01	0.80$\pm$0.01
cG+CN	0.86$\pm$0.03	0.85$\pm$0.02	0.86$\pm$0.03	0.85$\pm$0.03	0.79$\pm$0.02	0.77$\pm$0.02	0.80$\pm$0.01	0.77$\pm$0.02
cG+D+CN	0.85$\pm$0.02	0.83$\pm$0.01	0.85$\pm$0.02	0.82$\pm$0.02	0.79$\pm$0.02	0.78$\pm$0.01	0.79$\pm$0.01	0.78$\pm$0.02
GAMO\D (Ours)	0.87$\pm$0.01	0.86$\pm$0.01	0.88$\pm$0.01	0.87$\pm$0.01	0.81$\pm$0.01	0.80$\pm$0.01	0.82$\pm$0.01	0.80$\pm$0.01
GAMO (Ours)	0.89$\pm$0.01	0.88$\pm$0.01	0.91$\pm$0.01	0.90$\pm$0.01	0.82$\pm$0.01	0.80$\pm$0.01	0.83$\pm$0.01	0.81$\pm$0.01

Table 2. Table 2: Comparison of classification performance on CIFAR10 and SVHN datasets.

Dataset	Algorithm	ACSA	GM
CIFAR10	Baseline CN	0.45$\pm$0.01	0.37$\pm$0.01
	Augment+CN	0.47$\pm$0.01	0.39$\pm$0.02
	cDCGAN+CN	0.42$\pm$0.02	0.32$\pm$0.03
	DOS	0.46$\pm$0.02	0.37$\pm$0.01
	GAMO\D (Ours)	0.47$\pm$0.01	0.40$\pm$0.01
	GAMO (Ours)	0.49$\pm$0.01	0.43$\pm$0.02
SVHN	Baseline CN	0.74$\pm$0.01	0.73$\pm$0.01
	Augment+CN	0.69$\pm$0.01	0.63$\pm$0.01
	cDCGAN+CN	0.69$\pm$0.01	0.66$\pm$0.02
	DOS	0.71$\pm$0.02	0.68$\pm$0.01
	GAMO\D (Ours)	0.75$\pm$0.01	0.75$\pm$0.02
	GAMO (Ours)	0.76$\pm$0.01	0.75$\pm$0.02

Table 3. Table 3: Comparison of classification performance with increased number of training instances on CelebA and LSUN datasets.

	CelebA-Small				CelebA-Large
Algorithm	During Training		During Testing		During Training		During Testing
	ACSA	GM	ACSA	GM	ACSA	GM	ACSA	GM
Baseline CN	0.91$\pm$0.01	0.91$\pm$0.01	0.59$\pm$0.01	0.45$\pm$0.04	0.93$\pm$0.01	0.92$\pm$0.01	0.71$\pm$0.01	0.60$\pm$0.03
Augment+CN	0.74$\pm$0.06	0.70$\pm$0.09	0.62$\pm$0.05	0.47$\pm$0.08	0.82$\pm$0.01	0.79$\pm$0.01	0.72$\pm$0.01	0.66$\pm$0.02
cDCGAN+CN	0.86$\pm$0.01	0.84$\pm$0.01	0.59$\pm$0.01	0.36$\pm$0.02	0.87$\pm$0.01	0.86$\pm$0.01	0.67$\pm$0.01	0.58$\pm$0.02
DOS	0.82$\pm$0.03	0.80$\pm$0.02	0.61$\pm$0.01	0.48$\pm$0.02	0.84$\pm$0.01	0.83$\pm$0.02	0.72$\pm$0.01	0.64$\pm$0.02
GAMO (Ours)	0.92$\pm$0.01	0.91$\pm$0.01	0.66$\pm$0.01	0.54$\pm$0.02	0.91$\pm$0.01	0.91$\pm$0.01	0.75$\pm$0.01	0.70$\pm$0.02
	LSUN-Small				LSUN-Large
	During Training		During Testing		During Training		During Testing
	ACSA	GM	ACSA	GM	ACSA	GM	ACSA	GM
Baseline CN	0.90$\pm$0.01	0.89$\pm$0.01	0.50$\pm$0.01	0.28$\pm$0.05	0.87$\pm$0.01	0.87$\pm$0.01	0.61$\pm$0.02	0.54$\pm$0.03
Augment+CN	0.67$\pm$0.06	0.64$\pm$0.09	0.54$\pm$0.03	0.45$\pm$0.07	0.70$\pm$0.03	0.65$\pm$0.03	0.64$\pm$0.02	0.58$\pm$0.03
cDCGAN+CN	0.80$\pm$0.02	0.79$\pm$0.02	0.53$\pm$0.02	0.43$\pm$0.03	0.81$\pm$0.02	0.80$\pm$0.02	0.60$\pm$0.02	0.53$\pm$0.03
DOS	0.78$\pm$0.03	0.76$\pm$0.02	0.54$\pm$0.02	0.44$\pm$0.02	0.79$\pm$0.02	0.77$\pm$0.02	0.63$\pm$0.02	0.61$\pm$0.03
GAMO (Ours)	0.93$\pm$0.01	0.93$\pm$0.01	0.57$\pm$0.01	0.50$\pm$0.02	0.80$\pm$0.01	0.80$\pm$0.01	0.70$\pm$0.02	0.68$\pm$0.03

Table 4. Table 4: Comparison of classification performance on SUN397.

Algorithm	ACSA	GM
Baseline CN	0.26$\pm$0.04	0.19$\pm$0.05
Augment+CN	0.30$\pm$0.04	0.21$\pm$0.04
cDCGAN+CN	0.20$\pm$0.05	0.00$\pm$0.00
DOS	0.28$\pm$0.04	0.20$\pm$0.05
GAMO (Ours)	0.32$\pm$0.04	0.24$\pm$0.03

Table 5. Table 5: Comparison of FID of cDCGAN and GAMO2pix.

Dataset	GAMO2pix (Ours)	cDCGAN
Fashion-MNIST	0.75$\pm$0.03	5.57$\pm$0.03
SVHN	0.17$\pm$0.02	0.59$\pm$0.04
CIFAR10	1.59$\pm$0.03	2.96$\pm$0.03
CelebA-Small	11.13$\pm$0.04	15.12$\pm$0.05

Equations

$$G(\mathbf{z}|i)=\sum _{j=1}^{n_i}{g}_i(t(\mathbf{z}|i)){\mathbf{x}}_j,$$

$$\underset{G}{\min }\underset{M}{\max }J(G,M)=\sum _{i\in \mathcal{C}}{J}_i,$$

$$\text{where }{J}_i=(J_{i1}+J_{i2}+J_{i3}+J_{i4}),$$

$$J_{i1}=P_i{\mathbb{E}}_{\mathbf{x}\sim p_i^d}[\log M_i(\mathbf{x})],$$

$$J_{i2}=\sum _{j\in \mathcal{C}\setminus \{i\}}{P}_j{\mathbb{E}}_{\mathbf{x}\sim p_j^d}[\log (1-M_i(\mathbf{x}))],$$

$$J_{i3}=(P_c-P_i){\mathbb{E}}_{G(\mathbf{z}|i)\sim p_i^g}[\log M_i(G(\mathbf{z}|i))],\text{and},$$

$$J_{i4}=\sum _{j\in \mathcal{C}\setminus \{i\}}(P_c-P_j){\mathbb{E}}_{G(\mathbf{z}|j)\sim p_j^g}[\log (1-M_i(G(\mathbf{z}|j)))],$$

$$\sum_{i=1}^{c}JS\Big{(}\big{(}P_{i}p_{i}^{d}+(P_{c}-P_{i})p_{i}^{g}\big{)}\Big{|}\Big{|}\sum_{\begin{subarray}{c}j\neq i\\ j=1\end{subarray}}^{c}\big{(}P_{j}p_{j}^{d}+(P_{c}-P_{j})p_{j}^{g}\big{)}\Big{)}$$

$$\underset{G}{\min }\underset{M}{\max }\underset{D}{\max }Q(G,M,D)=\sum _{i\in \mathcal{C}}{Q}_i,$$

$$\text{where, }{Q}_i=(J_{i1}+J_{i2}+J_{i3}+J_{i4}+Q_{i1}+Q_{i2}),$$

$$Q_{i1}=P_i{\mathbb{E}}_{\mathbf{x}\sim p_i^d}[\log D(\mathbf{x}|i)],\text{and},$$

$$Q_{i2}=(P_c-P_i){\mathbb{E}}_{G(\mathbf{z}|i)\sim p_i^g}[\log (1-D(G(\mathbf{z}|i)|i))].$$

$$\underset{M}{\min }{L}_M=\sum _{i\in \mathcal{C}}(L_{i1}+L_{i2}+L_{i3}+L_{i4}),$$

$$\underset{D}{\min }{L}_D=\sum _{i\in \mathcal{C}}(L_{i5}+L_{i6}),$$

$$\underset{G}{\min }{L}_G=\sum _{i\in \mathcal{C}\setminus \{c\}}(L_{i7}+L_{i8}+L_{i9}),$$

$$\text{where, }{L}_{i1}=P_i{\mathbb{E}}_{\mathbf{x}\sim p_i^d}[(1-M_i(\mathbf{x}){)}^2],$$

$$L_{i2}=\sum _{j\in \mathcal{C}\setminus \{i\}}{P}_j{\mathbb{E}}_{\mathbf{x}\sim p_j^d}[(M_i(\mathbf{x}){)}^2],$$

$$L_{i3}=(P_c-P_i){\mathbb{E}}_{G(\mathbf{z}|i)\sim p_i^g}[(1-M_i(G(\mathbf{z}|i)){)}^2],$$

$$L_{i4}=\sum _{j\in \mathcal{C}\setminus \{i\}}(P_c-P_j){\mathbb{E}}_{G(\mathbf{z}|j)\sim p_j^g}[(M_i(G(\mathbf{z}|j)){)}^2],$$

$$L_{i5}=P_i{\mathbb{E}}_{\mathbf{x}\sim p_i^d}[(1-D(\mathbf{x}|i){)}^2],$$

$$L_{i6}=(P_c-P_i){\mathbb{E}}_{G(\mathbf{z}|i)\sim p_i^g}[(D(G(\mathbf{z}|i)|i){)}^2],$$

$$L_{i7}={\mathbb{E}}_{G(\mathbf{z}|i)\sim p_i^g}[(M_i(G(\mathbf{z}|i)){)}^2],$$

$$L_{i8}=\sum _{j\in \mathcal{C}\setminus \{i,c\}}{\mathbb{E}}_{G(\mathbf{z}|j)\sim p_j^g}[(1-M_i(G(\mathbf{z}|j)){)}^2],\text{and},$$

$$L_{i9}={\mathbb{E}}_{G(\mathbf{z}|i)\sim p_i^g}[(1-D(G(\mathbf{z}|i)|i){)}^2].$$

Full text

1 Introduction

The problem of class imbalance occurs when all the classes present in a dataset do not have equal number of representative training instances [19, 11]. Most of the existing learning algorithms produce inductive bias favoring the majority class in presence of class imbalance in the training set, resulting in poor performance on the minority class(es). This is a problem which routinely plagues many real-world applications such as fraud detection, dense object detection [30], medical diagnosis, etc. For example, in a medical diagnosis application, information about unfit patients is scarce compared to that of fit individuals. Hence, traditional classifiers may misclassify some unfit patients as being fit, having catastrophic implications [32].

Over the years, the machine learning community has devised many methods for tackling class imbalance [24, 4]. However, only a few of these techniques have been extended to deep learning even though class imbalance is fairly persistent is such networks, severely affecting both the feature extraction as well as the classification process [48, 21, 49, 5, 22]. The existing solutions [21, 10, 43, 30, 6] for handling class imbalance in deep neural networks mostly focus on cost tuning to assign suitably higher costs to minority instances. Another interesting class of approaches [50, 12] focuses on constructing balanced subsamples of the dataset. Wang et al. [44] proposed a novel meta-learning scheme for imbalanced classification. It is interesting to note that oversampling techniques like SMOTE [8] have not received much attention in the context of deep learning, despite being very effective for classical systems [14]. This is because deep feature extraction and classification are performed in an end-to-end fashion, making it hard to incorporate oversampling which is typically done subsequent to feature extraction. An attempt to bridge this gap was made by Ando and Huang [1] in their proposed deep oversampling framework (DOS). However, DOS uniformly oversamples the entire minority class and is not capable of concentrating the artificial instances in difficult regions. Additionally, the performance of DOS depends on the choice of the class-wise neighborhood sizes, which must be determined by costly parameter tuning.

Generative adversarial networks (GANs) are a powerful subclass of generative models that have been successfully applied to image generation. This is due to their capability to learn a mapping between a low-dimensional latent space and a complex distribution of interest, such as natural images [15, 33, 36, 35]. The approach is based on an adversarial game between a generator that tries to generate samples which are similar to real samples and a discriminator that tries to discriminate between real training samples and generated samples. The success of GANs as generative models has led Douzas and Bacao [13] to investigate the utility of using GANs to oversample the minority class(es). However, attempting to oversample the minority class(es) using GANs can lead to boundary distortion [39], resulting in a worse performance on the majority class (as illustrated in Figure 1(b)). Moreover, the generated points are likely to lie near the mode(s) of the minority class(es) [42], while new points around the class boundaries are required for learning reliable discriminative (classification) models [17, 18].

Hence, in this article, we propose (in Section 3) a novel end-to-end feature-extraction-classification framework called Generative Adversarial Minority Oversampling (GAMO) which employs adversarial oversampling of the minority class(es) to mitigate the effects of class imbalance. The contributions made in this article differ from the existing literature in the following ways:

Unlike existing deep oversampling schemes [1, 13], GAMO is characterized by a three-player adversarial game among a convex generator $G$, a classifier network $M$, and a discriminator $D$. 2. 2.

Our approach is fundamentally different from existing adversarial classification schemes (where the generator works in harmony with the classifier to fool the discriminator) [38, 27, 41, 35], in that our convex generator $G$ attempts to fool both $M$ and $D$. 3. 3.

Unlike the generator employed in GAN [15], we constrain $G$ to conjure points within the convex hull of the class of interest. Additionally, the discriminator $D$ further ensures that $G$ adheres to the class distribution for non-convex classes. Consequently, the adversarial contention with $M$ pushes the conditional distribution(s) learned by $G$ towards the periphery of the respective class(es), thus helping compensate for class imbalance effectively. 4. 4.

In contrast to methods like [8, 13], $G$ can oversample different localities of the data distribution to different extents based on the gradients obtained from $M$. 5. 5.

For applications requiring a balanced training set of images, we also propose a technique called GAMO2pix (Section 5) that can generate realistic images from the synthetic instances generated by GAMO in the distributed representation space.

We undertake an ablation study as well as evaluate the performance of our method compared to the state-of-the-art in Section 4, and make concluding remarks in Section 6.

2 Related Works

The success of SMOTE [8, 9] has inspired several improvements. For example, [17, 7] attempt to selectively oversample minority class points lying close to the class boundaries. Works like [18, 29, 2], on the other hand, asymmetrically oversample the minority class such that more synthetic points are generated surrounding the instances which are difficult to classify. Although these methods achieved commendable improvement on classical classifiers, they can neither be extended to deep learning techniques nor be applied to images, respectively due to the end-to-end structure of deep learning algorithms and a lack of proper notion of distance between images.

Extending GANs for semi-supervised learning, works like [27, 38] fused a $c$-class classifier with the discriminator by introducing an extra output line to identify the fake samples. On the other hand, [41] proposed a $c$-class discriminator which makes uncertain predictions for fake images. Additionally, [35] proposed a shared discriminator-cum-classifier network which makes two separate sets of predictions using two different output layers. These approaches can loosely be considered to be related to GAMO as these also incorporate a classifier into the adversarial learning scheme.

3 Proposed Method

Let us consider a $c$-class classification problem with a training dataset $X\subset\mathbb{R}^{D}$ (of images vectorized either by flattening or by a convolutional feature extraction network $F$). Let the prior probability of the $i$-th class be $P_{i}$, where $i\in\mathcal{C}=\{1,2,\cdots c\}$; $\mathcal{C}$ being the set of possible class labels. Without loss of generality, we consider the classes to be ordered such that $P_{1}\leq P_{2}\leq\cdots<P_{c}$. Moreover, let $X_{i}$ denote the set of all $n_{i}$ training points which belong to class $i\in\mathcal{C}$. We intend to train a classifier $M$ having $c$ output lines, where the $i$-th output $M_{i}(\mathbf{x})$ predicts the probability of any $\mathbf{x}\in X$ to be a member of the $i$-th class.

3.1 Adversarial Oversampling

Our method plays an adversarial game between a classifier that aims to correctly classify the data points and a generator attempting to spawn artificial points which will be misclassified by the classifier. The idea is that generating such difficult points near the fringes of the minority class(es) will help the classifier to learn class boundaries which are more robust to class imbalance. In other words, the performance of the classifier will adversarially guide the generator to generate new points at those regions where the minority class under concern is prone to misclassification. Moreover, the classifier will aid the generator to adaptively determine the concentration of artificial instances required to improve the classification performance in a region, thus relieving the user from tuning the amount of oversampling. Instead, we only need to fix the number of points to be generated to the difference between the number of points in the majority class and that of the (respective) minority class(es).

3.2 Convex Generator

The generator tries to generate points which will be misclassified by the classifier. Hence, if left unchecked, the generator may eventually learn to generate points which do not coincide with the distribution of the intended minority class. This may help improve the performance on the concerned minority class but will lead to high misclassification from the other classes. To prevent this from happening, we generate the new points only as convex combinations of the existing points from the minority class in question. This will restrict the generated distribution within the convex hull of the real samples from the (respective) minority class(es). Since the generator attempts to conjure points that are difficult for the classifier, the points are generated near the peripheries of the minority class(es).

Our convex generator $G$ comprises of two modules: a Conditional Transient Mapping Unit ($cTMU$) and a set of class-specific Instance Generation Units ($IGU$), which we propose to limit the model complexity. The $cTMU$ network learns a mapping $t$, conditioned on class $i$, from a $l$-dimensional latent space to an intermediate space. The $IGU_{i}$, on the other hand, learns a mapping $g_{i}$ from the $cTMU$ output space to a set of $n_{i}$ convex weights $g_{i}(t(\mathbf{z}|i))\geq 0$, s.t. $\sum_{j=1}^{n_{i}}g_{i}(t(\mathbf{z}|i))=1$, using softmax activation. Hence, $G$ can generate a new $D$-dimensional sample for the $i$-th class as a convex combination of the data points in $X_{i}$,

[TABLE]

where $\mathbf{z}$ is a latent variable drawn from a standard normal distribution and $\mathbf{x}_{j}\in X_{i}$.

Formally, the adversarial game played by the proposed classifier-convex generator duo poses the following optimization problem, when cross entropy loss is considered:

[TABLE]

while $p^{d}_{i}$ and $p^{g}_{i}$ respectively denote the real and generated class conditional probability distributions of the $i$-th class.

The two-player minimax game formalized in (2) is played between a classifier $M$ and a generator $G$. $M$ attempts to correctly classify all real as well as generated points belonging to all the classes. Whereas, $G$ strives to generate sample(s) which have a high probability of being classified by $M$ into all other classes. To demonstrate how such an adversarial game can aid $M$ to learn a better class boundary, we illustrate its chronological progression in a more explanatory manner in Figure 2. In Theorem 1, we show that the optimization problem in (2) is equivalent to minimizing a sum of the Jensen-Shannon divergences.

Theorem 1**.**

Optimizing the objective function $J$ is equivalent to the problem of minimizing the following summation of Jensen-Shannon divergences:

[TABLE]

Proof.

See the supplementary document. ∎

The behavior of the proposed approach can be understood by interpreting Theorem 1. The optimization problem aims to bring the generated distribution, for a particular class, closer to the generated as well as real distributions for all other classes. Since the real distributions are static for a fixed dataset, the optimization problem in Theorem 1 essentially attempts to move the generated distributions for each class closer to the real distributions for all other classes. This is likely to result in the generation of ample points near the peripheries, which are critical to combating class imbalance. While doing so, the generated distributions for all classes also strive to come closer to each other. However, the generated distributions for the different classes do not generally collapses upon each other, being constrained to remain within the convex hulls of the respective classes.

3.3 Additional Discriminator

While the generator only generates points within the convex hull of the samples from the minority class(es), the generated points may still be placed at locations within the convex hull which do not correspond to the distribution of the intended class (recall Figure 1(c)). This is likely to happen if the intended minority class(es) are non-convex in shape. Moreover, we know from Theorem 1 that the generated distributions for different minority classes may come close to each other if the respective convex hulls overlap. To solve this problem, we introduce an additional conditional discriminator which ensures that the generated points do not fall outside the actual distribution of the intended minority class(es). Thus, the final adversarial learning system proposed by us consists of three players, viz. a multi-class classifier $M$, a conditional discriminator $D$ which given a class aims to distinguish between real and generated points, and a convex generator $G$ that attempts to generate points which, in addition to being difficult for $M$ to correctly classify, are also mistaken by $D$ to be real points sampled from the given dataset. The resulting three-player minimax game is formally presented in (3).

[TABLE]

3.4 Least-Square Formulation

Mao et al. [31] showed that replacing the popular cross entropy loss in GAN with least square loss can not only produce better quality images but also can prevent the vanishing gradient problem to a greater extent. Therefore, we also propose a variant of GAMO using the least square loss, which poses the following optimization problem:

[TABLE]

3.5 Putting it all together

The model for the GAMO framework is detailed in Figure 3, while the complete algorithm is described in Algorithm 1. To ensure an unbiased training for $M$ and $D$ we generate artificial points for the $i$-th class with probability $(P_{c}-P_{i})$ to compensate for the effect of imbalance. On the other hand, to also ensure unbiased training for $G$ we use samples from all classes with equal probability.

4 Experiments

We evaluate the performance of a classifier in terms of two indices which are not biased toward any particular class [40], namely Average Class Specific Accuracy (ACSA) [21, 44] and Geometric Mean (GM) [26, 4]. All our experiments have been repeated 10 times to mitigate any bias generated due to randomization and the means and standard deviations of the index values are reported. Codes for the proposed methods are available at https://github.com/SankhaSubhra/GAMO.

We have used a collection of 7 image datasets for our experiments, namely MNIST [28], Fashion-MNIST [46], CIFAR10 [25], SVHN [34], LSUN [51] and SUN397 [47]. All the chosen datasets except SUN397 are not significantly imbalanced in nature, therefore we have created their imbalanced variants by randomly selecting a disparate number of samples from the different classes. Further, for all the datasets except SUN397, 100 points are selected from each class to form the test set. In the case of SUN397 (50 classes of which are used for our experiments) 20 points from each class are kept aside for testing.

We refrain from using pre-trained networks for our experiments as the pre-learned weights may not reflect the imbalance between the classes. We, instead, train the models from scratch to emulate real-world situations where the data is imbalanced and there is no pre-trained network available that can be used as an appropriate starting point. We have obtained the optimal architectures and hyperparameters for each contending method in Section 4-5 using a grid search (see supplementary document).

4.1 MNIST and Fashion-MNIST

The experiments in this section are conducted using imbalanced subsets of the MNIST and Fashion-MNIST datasets. In case of both the datasets, we have sampled $\{4000,2000,1000,750,500,350,200,100,60,40\}$ points from classes in order of their index. Thus, the datasets have an Imbalance Ratio (IR: ratio of the number of representatives from the largest class to that of the smallest class) of 100. We begin by establishing the effectiveness of our proposed framework. We also compare between the two variants of GAMO which use Cross Entropy (CE) and Least Square (LS) losses, respectively.

We undertake an ablation study on MNIST using flattened images to facilitate straightforward visualization of the oversampled instances. Convolutional features are used for Fashion-MNIST. For MNIST, we have compared GAMO, against baseline classifier network (CN), SMOTE+CN (training set is oversampled by SMOTE), cGAN+CN (training set oversampled using cGAN, which is then used to train CN), and also traced the evolution of the philosophy behind GAMO, through cG+CN (conditional generator cG adversarially trained against CN, in contrast to cGAN+CN where CN does not play any part in training cGAN), cG+D+CN (cG+CN network coupled with a discriminator D), and GAMO\D (GAMO without a discriminator) on the MNIST dataset. SMOTE+CN and cGAN+CN are respectively replaced by Augment+CN (data augmentation is used to create new images for balancing the training sets), and cDCGAN+CN (oversampled using conditional deep convolutional GAN) for Fashion-MNIST. GAMO is also compared with DOS on Fashion-MNIST.

The ablation study is shown visually in Figure 4 and the results for both datasets are tabulated in Table 1. Overall, GAMO is observed to perform better than all other methods on both datasets. Interestingly, GAMO\D performs much worse than GAMO on MNIST but improves significantly on Fashion-MNIST. This may be due to the fact that the convolutional feature extraction for Fashion-MNIST results in distributed representations where the classes are almost convex with little overlap between classes, enabling the convex generator to always generate data points which reside inside the class distributions.

Since we observe from Table 1 that the LS variants of the classifiers mostly perform better than their CE based counterparts (which according to [31] is contributed by the more stable and better decision boundary learned in LS), all the experiments in the subsequent sections are reported using the LS formulation for all the contending algorithms.

4.2 CIFAR10 and SVHN

In case of CIFAR10 and SVHN the classes are subsampled (4500, 2000, 1000, 800, 600, 500, 400, 250, 150, and 80 points are selected in order of the class labels) to achieve an IR of 56.25. From Table 2 we can see that GAMO performs better than others on both of these datasets, closely followed by GAMO\D, further confirming the additional advantage of convolutional feature extraction in the GAMO framework. Interestingly, Augment+CN performs much worse than the other methods on the SVHN dataset. This may be due to the nature of the images in the SVHN dataset, which may contain multiple digits. In such cases, attempting to augment the images may result in a shift of focus from one digit to its adjacent digit, giving rise to a discrepancy with the class labels.

4.3 CelebA and LSUN

The experiment on CelebA and LSUN are undertaken to evaluate the performance of GAMO on images of higher resolution, as well as to assess the effects of an increase in the number of instances from the different classes. In case of CelebA the images are scaled to $64\times 64$ size, while for LSUN the same is done on a central patch of resolution $224\times 224$ extracted from each image. In the case of CelebA we have created two 5 class datasets by selecting samples from non-overlapping classes of hair colors, namely blonde, black, bald, brown, and gray. The first dataset is the smaller one (having 15000, 1500, 750, 300, and 150 points in the respective classes) with an IR of 100, while the second one is larger (having 28000, 4000, 3000, 1500, and 750 points in the respective classes) with an IR of 37.33. Similarly, in the case of LSUN we select 5 classes namely classroom, church outdoor, conference room, dining room, and tower, and two datasets are created. The smaller one (with 15000, 1500, 750, 300, and 150 points from the respective classes) has an IR of 100, while the larger one (with 50000, 5000, 3000, 1500, and 750 points) has an IR of 66.67.

In Table 3, we present the ACSA and GM over both the training and test set for the small and large variants of the two datasets. We can observe that all the algorithms manage to close the gap between their respective training and testing performances as the size of the dataset increases. Moreover, while Augment+CN seems to have the lowest tendency to overfit (smallest difference between training and testing performances), GAMO exhibits a greater ability to retain good performance on the test dataset.

4.4 SUN397

We have randomly selected 50 classes from SUN397 to construct a dataset containing $64\times 64$ sized images (depending on the image size either a $512\times 512$ or a $224\times 224$ center patch is extracted, which is then scaled to $64\times 64$) with an IR of 14.21. The experiment on SUN397 is performed to evaluate the performance of GAMO over a large number of classes. A scrutiny of the result tabulated in Table 4 reveals that despite all four contending techniques being severely affected by the complexity of the classes and the scarcity of data samples from many of the classes, GAMO is able to retain overall better performance than its competitors.

5 GAMO2pix

GAMO results ultimately in a classifier trained to properly classify samples from all the classes. However, some application may require that actual samples be generated by oversampling to form an artificially balanced dataset. While GAMO directly generates images if flattened images are used, it only generates vectors in the distributed representation space (mapped by the convolutional layers) for the convolutional variant. Therefore, we also propose the GAMO2pix mechanism to obtain images from the GAMO-generated vectors in the distributed representation space.

Our network for generating images (as illustrated in Figure 5(a)) from the GAMO-generated vectors is inspired by the Variational Autoencoder (VAE) [23, 37]. VAE, unlike regular autoencoders, is a generative model which attempts to map the encoder output to a standard normal distribution in the latent space, while the decoder is trained to map samples from the latent normal distribution to images. We follow the design of a standard VAE in GAMO2pix, only replacing the encoder part with the convolutional feature extractor $F$ trained by GAMO. The GAMO2pix network is trained separately for each class while keeping the encoder part fixed. Such a setting should learn the inverse map from the $D$-dimensional feature space induced by $F$ to the original image space and consequently be able to generate realistic images of the concerned class given GAMO-generated vectors for that class as input.

We present the images respectively generated by cDCGAN and GAMO2pix on CIFAR10, Fashion-MNIST, SVHN and CelebA-Small in Figures 5(b)-5(c). We can see that GAMO2pix can indeed generate more realistic and diverse images, compared to cDCGAN which also suffers from mode collapse for minority classes. This is further confirmed by the lower Fréchet Inception Distance (FID) [20] (calculated between real and artificial images from each class and averaged over classes) achieved by GAMO2pix, as shown in Table 5.

6 Conclusions and Future Work

The proposed GAMO is an effective end-to-end oversampling technique for handling class imbalance in deep learning frameworks. Moreover, it is also an important step towards training robust discriminative models using adversarial learning. We have observed from our experiments that the convolutional variant of GAMO is more effective due to the distributed representations learned by the convolutional layers. We also found that the LS loss variant of GAMO generally performs better than the CE loss variant.

An interesting area of future investigation is to improve the quality of the images generated by GAMO2pix by employing a different architecture such as BEGAN [3]. To reduce the tendency of GAMO to overfit as well as to potentially improve its performance, one may consider hybridization with improved GAN variants [16] which can achieve good performance even with less number of training samples. Further, one may explore the efficacy of GAMO to learn new classes by taking inspiration from Memory Replay GAN [45], or study the usefulness of the proposed convex generator for handling boundary distortion in GANs.