SeGMA: Semi-Supervised Gaussian Mixture Auto-Encoder

TL;DR

SeGMA introduces a semi-supervised Wasserstein auto-encoder with a Gaussian mixture latent space, enabling effective clustering, interpolation, style transfer, and class characteristic manipulation in generated data.

Contribution

It presents a novel semi-supervised generative model using Gaussian mixtures in latent space with efficient optimization and enhanced data manipulation capabilities.

Findings

Achieves comparable or better generative performance on benchmarks.

Enables realistic interpolation and style transfer between classes.

Allows manipulation of class characteristics by latent space movement.

Abstract

We propose a semi-supervised generative model, SeGMA, which learns a joint probability distribution of data and their classes and which is implemented in a typical Wasserstein auto-encoder framework. We choose a mixture of Gaussians as a target distribution in latent space, which provides a natural splitting of data into clusters. To connect Gaussian components with correct classes, we use a small amount of labeled data and a Gaussian classifier induced by the target distribution. SeGMA is optimized efficiently due to the use of Cramer-Wold distance as a maximum mean discrepancy penalty, which yields a closed-form expression for a mixture of spherical Gaussian components and thus obviates the need of sampling. While SeGMA preserves all properties of its semi-supervised predecessors and achieves at least as good generative performance on standard benchmark data sets, it presents additional features: (a) interpolation between any pair of points in the latent space produces realistically-looking samples; (b) combining the interpolation property with disentangled class and style variables, SeGMA is able to perform a continuous style transfer from one class to another; (c) it is possible to change the intensity of class characteristics in a data point by moving the latent representation of the data point away from specific Gaussian components.

Tables

Table 1. TABLE I: FID scores on various datasets (lower is better). The results in brackets for SVHN was obtained using convolutional architecture.

Variant	Model	MNIST	SVHN	CelebA
unsupervised	CWAE	20.76	49.30	51.94
	SeGMA	17.79	61.11	51.13
	Triple GAN	10.46	45.40	–
	M1+M2	40.50	142.11	70.96
semi-supervised	ADGM	14.50	55.54	51.96
	SeGMA	12.51	63.80 (50.10)	50.54

Table 2. TABLE II: Test errors on various datasets (lower is better). The results in brackets for SVHN was obtained using convolutional architecture.

Model	MNIST	SVHN	CelebA
Triple GAN	0.8%	5.8%	–
M1+M2	17.0%	65.7%	44.5%
ADGM	4.1%	41.2%	41.7%
SeGMA	12.7%	32.8% (22.4%)	24.6%

Table 3. TABLE III: FIDs of interpolations (lower is better). The results in brackets for SVHN was obtained using convolutional architecture.

Model	MNIST	SVHN	CelebA
M1+M2	56.38	176.89	88.24
ADGM	21.75	76.66	63.60
SeGMA	13.40	63.47 (50.01)	48.14

Equations

$$\mathrm{MSE}(X;\mathcal{E},\mathcal{D})=\frac{1}{n}\sum _{i=1}^n\Vert x_i-\mathcal{D}(\mathcal{E}(x_i)){\Vert }^2,$$

$$d_{\mathrm{MMD}}(\mathcal{E}X,P_{Z})\\ =\Big{\|}\int_{\mathcal{Z}}k(z,\cdot)\mathrm{d}P_{\mathcal{E}X}(z)-\int_{\mathcal{Z}}k(z,\cdot)\mathrm{d}P_{Z}(z)\Big{\|}_{\mathcal{H}_{k}},$$

$$\begin{gathered} ⟨N(a,\alpha I),N(b,\beta I){⟩}_{\gamma } \\ =\frac{1}{\sqrt{2\pi (\alpha +\beta +2\gamma )}}{\varphi }_D\left(\frac{\Vert a-b{\Vert }^2}{2(\alpha +\beta +2\gamma )}\right), \end{gathered}$$

$$d_{\gamma }^2(Z,P_Z)=⟨Z-P_Z,Z-P_Z{⟩}_{\gamma }$$

$$=\frac{1}{m^2}\sum _{i=1}^m\sum _{j=1}^m⟨N(z_i,0),N(z_j,0){⟩}_{\gamma }$$

$$-\frac{2}{m}\sum _{i=1}^m⟨N(z_i,0),N(0,I){⟩}_{\gamma }+⟨N(0,I),N(0,I){⟩}_{\gamma }$$

$$\mathcal{L}({\hat{P}}_{Y_L|\mathcal{E}{X}_L}\Vert P_{Y|Z})$$

$$=\frac{1}{m-n}\sum _{i=n+1}^m-\log P_{Y|Z}(y_i|\mathcal{E}(x_i))$$

$$=\frac{1}{m-n}\sum _{i=n+1}^m-\log \frac{P_Y(y_i)P_{Z|Y=y_i}(\mathcal{E}(x_i))}{P_Z(\mathcal{E}(x_i))}$$

$$\mathrm{MSE}(\hat{X};\mathcal{E},\mathcal{D})+\alpha d_{\mathrm{MMD}}(\mathcal{E}\hat{X},P_Z)+\beta \mathcal{L}({\hat{P}}_{Y_L|\mathcal{E}{X}_L}\Vert P_{Y|Z})$$

$$P_{Y|Z}(k|z)=\frac{p_{k}N({\mu }_k,{\Sigma }_k)(z)}{{\sum }_{l=1}^K{p}_{l}N({\mu }_l,{\Sigma }_l)(z)}.$$

$$d_{\gamma }^2(Z,P)=\frac{1}{m^2\sqrt{2\pi (2\gamma )}}\sum _{i=1}^m\sum _{j=1}^m{\varphi }_D\left(\frac{\Vert z_i-z_j{\Vert }^2}{4\gamma }\right)$$

$$-\sum _{i=1}^m\sum _{k=1}^K\frac{2p_k}{m\sqrt{2\pi ({\sigma }_k^2+2\gamma )}}{\varphi }_D\left(\frac{\Vert z_i-{\mu }_k{\Vert }^2}{2({\sigma }_k^2+2\gamma )}\right)$$

$$+\sum _{k=1}^K\sum _{l=1}^K\frac{p_k{p}_l}{\sqrt{2\pi ({\sigma }_k^2+{\sigma }_l^2+2\gamma )}}{\varphi }_D\left(\frac{\Vert {\mu }_k-{\mu }_l{\Vert }^2}{2({\sigma }_k^2+{\sigma }_l^2+2\gamma )}\right).$$

$$\begin{gathered} \underset{\mathcal{E},\mathcal{D},\{{\mu }_k\},\{p_k\}}{\min }\mathrm{MSE}(\hat{X};\mathcal{E},\mathcal{D}) \\ +\alpha \log d_{\gamma }^2(\mathcal{E}\hat{X},P_Z)+\beta \mathcal{L}({\hat{P}}_{Y_L|\mathcal{E}{X}_L}\Vert P_{Y|Z}), \end{gathered}$$

$$\hat{z}=z+({\mu }_t-{\mu }_s)$$

Full text

I Introduction

Deep generative models, such as the variational auto-encoder (VAE, [1]), Wasserstein auto-encoder (WAE, [2]), deep Boltzmann machines [3], or generative adversarial networks (GAN, [4]) play an important role in deep learning. They are capable of learning a probability distribution of complex high-dimensional data, which can be used to generate new examples. Their applications include image inpainting [5], 3D object reconstruction [6], sentence paraphrasing [7], and many more [8, 9, 10].

In more specialized applications where data is composed of classes, a natural question is how to build a generative model that is able to generate examples from particular classes. Since, in practice, the access to labeled data is limited, it is often not possible to train an individual generative model for each class using classical techniques. A step towards solving this problem was the introduction of semi-supervised versions of VAE models [11, 12]. Technically, these models learn an additional categorical, i.e., discrete latent variable for encoding class labels by making use of a small number of labeled samples together with a large collection of unlabeled data. While they still describe the whole data distribution as their unsupervised predecessors, they are also capable of generating samples from specific classes. These samples are produced by transforming the discrete variable describing the class together with a latent code generated from a continuous target distribution using a decoder network.

In this paper, we present an alternative approach for constructing generative models capable of creating class-specific samples. Instead of introducing a classification network with an additional discrete latent variable representing the class, we use a classical mixture of Gaussians to provide a natural disentanglement of class and style111Following [11], we decompose the information about each sample into two parts. The first factor includes the information needed to identify a class, while the second one accounts for variations within a class. This latter component is commonly referred to as style. information in a single latent space of the auto-encoder. Using a small labeled data set, classes are assigned to components of this mixture of Gaussians by minimizing the cross-entropy loss induced by the class posterior distribution of a simple Gaussian classifier. The resulting mixture describes the distribution of the whole data, and representatives of individual classes are generated by sampling from its components, see Fig. 1 for illustration.

Our model, which we will call semi-supervised Gaussian mixture auto-encoder (SeGMA), is an adaptation of the Cramer-Wold auto-encoder (CWAE) [13], which is an instance of WAE models with maximum mean discrepancy (MMD) penalty [2]. The use of CWAE yields a closed-form expression for the MMD penalty for spherical Gaussians, which eliminates the need of sampling and propagating gradients through discrete variables which is not possible in a straightforward manner [14, Sec. 1.1]. Experimental results on MNIST, SVHN, and CelebA data sets show that SeGMA obtains comparable or even better generative performance than corresponding VAE models. Moreover, it has several advantages, which cannot be obtained using previous models:

• •

Due to the use of a single continuous latent space in which the class information is encoded implicitly, we can interpolate between any two data points as in classical unsupervised generative models. In consequence, we are able to generate samples which mix properties of several classes. This property is difficult to obtain for semi-supervised VAE models, because they are trained directly on discrete class variables for which intermediate states are not necessarily meaningful.

• •

Despite lacking an architectural separation between the discrete class label and the continuous latent code, we demonstrate that SeGMA is able to disentangle the class information of a data point from its style. In contrast to semi-supervised VAE models, in which this disentanglement is explicit, SeGMA allows to perform continuous style transfer. In other words, we can change the class variable smoothly, while preserving the style of examples. This is a consequence of interpolation property, see Fig. 2. While continuous style transfer is performed by moving a latent representation from the original Gaussian component towards the component of the target class, the intensity of the style w.r.t. certain class-discriminating characteristics can be increased by moving the latent representation away from specific Gaussian components.

• •

SeGMA is easy to implement in an existing WAE-MMD framework and is robust to the choice of hyperparameters (see Section VI-E). We achieve better generative performance than related VAE approaches despite using simpler neural networks and significantly lower-dimensional latent space – e.g., we use a classical encoder-decoder network with a 20-dimensional latent space for SVHN, while the method proposed by [12] consists of six networks and 300 dimensions for encoding latent codes. SeGMA is trained faster and in a more stable way due to the use of Cramer-Wold distance which eliminates the need of sampling. Finally, since we use a classical auto-encoder architecture, it is easier to apply convolutional networks than in semi-supervised VAE models, where continuous and discrete inputs have to be gathered together.

To give a concrete real-life example where the proposed model is useful, let us consider a graphical application for modifying faces. More precisely, the application takes a face image and allows to modify its selected features, such as smile, type of beard, age, etc. Clearly, it should be possible to change the beard length, choose a gentle or wide smile as well as let the face age. We emphasize that it is desirable to be able to change these features smoothly, rather than having to choose among few discrete options. Our model satisfies these requirements by interpolating between features and changing the intensity of labels as can be seen in Fig. 2. For more details, see Sections VI-C and VI-D.

The remainder of this work is organized as follows: We discuss the similarities between our approach and the related works in semi-supervised and unsupervised generative models in Section II. The WAE and the Cramer-Wold distance are revised in Section III. We complete the theoretical framework of SeGMA in Section IV, where we include a regularization term based on a small labeled data set. For the specific choice of a mixture of Gaussians as a target distribution in latent space, we instantiate our theory in Section V and briefly discuss optimization issues. We conclude by showing experimental results and comparisons with state-of-the-art semi-supervised VAE models in Section VI.

II Related work

There are a few works on unsupervised deep clustering, which use a mixture of Gaussians as a target distribution in latent space. The authors of [15] propose a generative model based on the VAE in which both the encoder and decoder are stochastic.222We follow the literature and call a neural network (encoder/decoder) stochastic if the network parameterizes a probability distribution, i.e., if the network output is sampled from a distribution parameterized by the network parameters and the input. We call a neural network deterministic if the network parameterizes a function, i.e., if the network output is a function of the input parameterized by the network parameters. They train their model by maximizing the evidence lower bound (ELBO) over the parameters of the Gaussian mixture model and the parameters of the encoder and decoder. A very similar model was recently introduced in [16], with the main difference that the decoder is deterministic and that, instead of maximizing the ELBO, a variational bound on the information bottleneck functional is optimized, cf. [17], which is simultaneously an upper bound to the ELBO proposed in [15] for certain parameter settings, cf. [16, Remark 1].

Despite being unsupervised, both [15] and [16] report that the obtained clusters agree very well with the class labels. However, [15, 16], and also [17, 18], evaluated the generative performance of their models only on MNIST. For more complicated datasets, an agreement between clusters and classes need not occur. Indeed, [19], proposed a Gaussian mixture model in which the means and variances are learned by training the parameters of an additional neural network with a standard Gaussian distribution at its input. The authors show that the clusters coincide with the individual classes for MNIST, but tend to group images that are visually similar for SVHN, respectively; cf. [19, Figs. 5 & 6].

Another approach for training deep generative models relies on using WAE [2]. The authors of [20] show that in some cases the optimization of a classical shallow Gaussian mixture model using sliced p-Wasserstein distance [21] leads to better parameters than applying the EM algorithm. Instead of a joint optimization of Wasserstein distance with neural network weights, they fit a Gaussian mixture model on a fixed embedding of data. Reference [18] explains the difficulties in training VAE models with discrete latent variables [22, 23, 24]. As a remedy they propose a WAE-MMD model with a mixture of Gaussians as a target distribution. This fully unsupervised model uses discrete and continuous latent variables and is optimized by sampling from a target distribution.

The semi-supervised deep generative models proposed in the literature differ substantially from the unsupervised generative models discussed so far. The most important difference is that they have two explicit latent variables, a discrete one accounting for the class and a continuous one, commonly attributed to represent the style of the data point. Thus, these methods explicitly consider the discrete variable also during inference, e.g., by training a classifier network on the latent space in addition to an encoder network. For example, [11] proposed a generative model parametrized by a stochastic neural network taking the latent class and style variables as inputs. Similarly, encoder and classifier – computing the style and class variables, respectively – are implemented as stochastic neural networks. All model parameters are optimized by maximizing the ELBO over the unlabeled and labeled parts of the data set; for the former, the class variable is supplied via posterior inference using the classifier network. The authors of [12] improved upon [11] by adding a continuous auxiliary latent variable, thereby increasing the capacity of the model and improving the variational bound. By using neural networks for class inference, both [11] and [12] achieve good classification and generative performance with only a small labeled data set. The paper [25] follows a different approach and modifies the VAE model so that one of the continuous latent variables represents a clustering structure of the data. The authors of [26] build their semi-supervised model on GANs, but they focus directly on a classification task. They show that a generator that models the data distribution perfectly neither leads to a better classifier nor utilizes unlabeled data effectively. To achieve good classification performance, they thus propose a generator that does not produce realistic samples.

A relevant example from supervised learning is [27]. Combining the loss function of the VAE with a regularizer that rewards latent codes that are independent of the class variable, the authors obtain a generative model that creates MNIST images from the latent code (thus representing only style) and a discrete class variable that they feed as an additional input.

On the one hand, our setting is conceptually similar to generative deep clustering approaches which use mixtures of Gaussians as target distributions. Most of these works use a single continuous latent space (except the WAE-MMD approach [18], which uses two latent variables), but require sampling from some distribution during training. In contrast, the use of the Cramer-Wold distance in SeGMA allows fitting the encoded data to the target mixture using a closed-form expression. Furthermore, our focus is on the generative properties rather than on clustering. In comparison, [15, 18] generated samples only for MNIST, [16] did not generate samples at all, and none of the mentioned works evaluate the quality of their samples using quantitative methods.

On the other hand, our work draws concepts from semi-supervised VAEs, which are comprised of (at least) two latent spaces, with one of them being discrete. The main difference to semi-supervised VAEs is that we propose using a single continuous latent space that allows us not only to transfer style to different classes, but also to interpolate between data points from different classes. Furthermore, we eliminate the need for an auxiliary classifier network by simply computing the loss of a Gaussian classifier based on a small labeled data set. Finally, our cost function is not based on variational principles, but is motivated by WAEs.

III Preliminaries: Wasserstein Auto-Encoder and Cramer-Wold Distance

Our approach is based on the WAE with a MMD penalty [2]. As opposed to the VAE [1], WAE allows using deterministic, i.e., non-random, encoders mapping the input to a latent space via a function. To eliminate the need for sampling during training and to improve the model stability, we implement MMD using Cramer-Wold distance [13].

Let $X=(x_{i})_{i=1}^{n}\subset\mathbb{R}^{N}$ be a data set. A non-random auto-encoder consists of a deterministic encoder $\mathcal{E}{:}\ \mathbb{R}^{N}\to\mathcal{Z}$ and a deterministic decoder $\mathcal{D}{:}\ \mathcal{Z}\to\mathbb{R}^{N}$, where $\mathcal{Z}=\mathbb{R}^{D}$ is a latent space (usually $D<N$). The encoder thus provides a lower-dimensional representation of $X$ that the decoder shall be able to reconstruct with as small an error as possible. This goal is achieved by minimizing the mean squared error between input and output of the auto-encoder:

[TABLE]

where $\|\cdot\|$ denotes the Euclidean norm.

To make the auto-encoder generative, the reconstruction error is complemented by a regularization term that encourages the distribution of the low-dimensional representation $\mathcal{E}X$ to be similar to a target distribution $P_{Z}$. Specifically, for a WAE with MMD penalty, the regularization term is chosen as the norm in some reproducing kernel Hilbert space $\mathcal{H}_{k}$, i.e.,

[TABLE]

where $P_{\mathcal{E}X}$ is the distribution of $\mathcal{E}X$ and where $k{:}\ \mathcal{Z}\times\mathcal{Z}\to\mathbb{R}$ is a positive definite reproducing kernel. If the distribution of $\mathcal{E}X$ is similar to $P_{Z}$ and if $\mathcal{E}$ and $\mathcal{D}$ are such that $\mathrm{MSE}(X;\mathcal{E},\mathcal{D})$ is small, then samples from $X$ can be generated by sampling from $P_{Z}$ and passing these samples to the decoder $\mathcal{D}$ [13, Remark 4.1].

During the training process, the comparison of the latent data representations $\mathcal{E}X$ with a target distribution $P_{Z}$ is performed by sampling from $P_{Z}$ and approximating (2) by averaging the kernel $k$ evaluated at all pairs of samples [2, Algorithm 2]. Such a procedure is necessary even for a simple target distribution such as the standard Gaussian $N(0,I)$. To eliminate the need for sampling, MMD can be implemented using the Cramer-Wold distance, which has a closed-form expression for spherical Gaussian distributions $N(\mu,\sigma^{2}I)$. This approach results in Cramer-Wold auto-encoder (CWAE) [13], which is an instance of WAE-MMD with inverse multiquadratics kernel [2, page 9].

The Cramer-Wold distance $d^{2}_{\gamma}$, where $\gamma>0$ is a regularization parameter, is induced by a corresponding scalar product $\langle\cdot,\cdot\rangle_{\gamma}$, which for two spherical, $D$-dimensional Gaussians $N(a,\alpha I),N(b,\beta I)$ is given by [13, eq. 11]:

[TABLE]

where $\phi_{D}(s)\approx(1+\tfrac{4s}{2D-3})^{-1/2}$, for $D\geq 10$.

To evaluate (3) for a sample $Z=(z_{i})_{i=1}^{m}$, we represent it as the uniform probability measure on this sample $\frac{1}{m}\sum_{i=1}^{m}\delta_{z_{i}}$, where $\delta_{z}$ is an atom Dirac measure condensed at $z$. Note that a Dirac measure $\delta_{z}$ can be interpreted as a degenerate Gaussian density333A degenerate Gaussian density has a singular (non-invertible) covariance matrix, see [28]. with mean $z$ and the zero matrix as covariance matrix444Such a Gaussian density is spanned on 0-dimensional linear subspace.. Consequently, a sample $Z=(z_{i})_{i=1}^{m}$ is represented as the uniform mixture of degenerate Gaussians with zero covariance, i.e. $Z=\frac{1}{m}\sum_{i=1}^{m}N(z_{i},0)$. If we put $P_{Z}=N(0,I)$, then the Cramer-Wold distance between this target distribution and the sample $Z$ equals:

[TABLE]

where $\langle\cdot,\cdot\rangle_{\gamma}$ is given by (3). The parameter $\gamma$ can be seen as a kernel bandwidth and is commonly selected using Silverman’s rule of thumb for Gaussian densities.

IV Theoretical Model: Semi-Supervised Learning for Improved Class Representation

Classical WAE models allow us to describe the distribution of the data and to generate samples from it. In practice, data can come from several distinct classes. In such a case, we want to model a distribution of the data as a whole as well as to be able to generate samples from a given class.

To this end, we introduce a joint distribution $P_{Z,Y}$, which additionally encodes the latent class variable $Y$. On the one hand, marginalization over the class variable $Y$ yields the target distribution $P_{Z}$ to which the WAE fits the latent representation. On the other hand, to generate samples from a given class $y$, we fix this class in the distribution $P_{Z,Y}$, sample from the resulting conditional distribution $P_{Z|Y=y}$, and pass the obtained latent codes through the decoder $\mathcal{D}$. Thus, $P_{Z,Y}$ should be parametrized such that both:

• •

samples from its marginal $P_{Z}$ have high quality,

• •

samples from $P_{Z|Y=y}$ are representatives of class $y$.

In practical cases, $P_{Z}$ could be a multi-modal distribution, with each mode corresponding to one class.

To parametrize $P_{Z,Y}$ (e.g., to assign the modes of the distribution $P_{Z}$ to individual classes), we follow a typical semi-supervised learning scenario and use a small number of labeled examples [29]. It is assumed that we have access to a labeled data set $(X_{L},Y_{L})=(x_{i},y_{i})_{i=n+1}^{m}$, where $X_{L}\subset\mathbb{R}^{N}$ and where $y_{i}\in\{1,\ldots,K\}$ denotes the class label. To incorporate knowledge coming from labeled examples in building our generative model, we use a classifier. More precisely, let $P_{Y|Z}$ be the class posterior probability and let $\hat{P}_{Y_{L},\mathcal{E}X_{L}}=\frac{1}{m-n}\sum_{i=n+1}^{m}\delta[(y,z)=(y_{i},\mathcal{E}x_{i})]$ be the empirical joint distribution of the labeled encoded data set, where $\delta[\cdot]$ is the Kronecker delta. We complement the WAE objective with the cross-entropy between the true class label and the class label inferred from $P_{Y|Z}$:

[TABLE]

This suggests that a good generative model in the proposed semi-supervised setting consists of an encoder $\mathcal{E}$ and a decoder $\mathcal{D}$ such that

[TABLE]

is minimized, where $\hat{X}=X\cup X_{L}$ and $\alpha,\beta$ are trade-off parameters.

V Practical Implementation

In this section, we present our choice for the distribution $P_{Z,Y}$, instantiate the cost function (6) for this choice and discuss its optimization.

In order to keep the target distribution simple, we choose $P_{Z,Y}$ to be a Gaussian mixture model. In this case, $P_{Y}$ is a categorical distribution from $K$ categories with prior probabilities $p_{1},\ldots,p_{K}$, while $P_{Z|Y=k}$ is a multivariate Gaussian $N(\mu_{k},\Sigma_{k})$, which describes the distribution of class $k$ in latent space. The marginalization of $P_{Z,Y}$ over classes $Y$ gives a target distribution $P_{Z}$ which is a mixture of Gaussians with density $P_{Z}=\sum_{k=1}^{K}p_{k}N(\mu_{k},\Sigma_{k})$.

On the one hand, choosing $P_{Z,Y}$ to be a Gaussian mixture model makes it easy to generate latent codes for a given class $k$ by sampling from the conditional density $P_{Z|Y=k}=N(\mu_{k},\Sigma_{k})$. At the same time, the posterior probability used in the semi-supervised setting simplifies to:

[TABLE]

Using such a simple Gaussian classifier in the feature space directly may fail; we show in our experiments that its embedding in the latent space yields satisfactory performance.

On the other hand, there appears the question how to compute $d_{\mathrm{MMD}}(\mathcal{E}X,P_{Z})$ for the mixture of Gaussians and how to train a network minimizing the WAE objective in this case. The simplest approach relies on sampling from the mixture density $P_{Z}$ and next computing the MMD distance between these samples and encoded data. The implementation of this step requires the reparameterization trick to back-propagate the gradients. However, since we have $K$ Gaussians, the number of samples should probably be $K$ times larger than for a single Gaussian to approximate this this distribution well enough, which might negatively influence the training time.

Our choice to implement the MMD penalty with the Cramer-Wold distance [13] resolves this issue since it obviates the need for sampling, effectively decreasing training time and improving model stability (see [13, Section 7.4] for a comparison between CWAE, WAE and VAE). Combining (3) and (III), the Cramer-Wold distance between a sample $Z=(z_{i})_{i=1}^{m}$ and a mixture of spherical Gaussians $P=\sum_{k=1}^{K}p_{k}N(\mu_{k},\sigma^{2}_{k}I)$ can be computed in closed form:

[TABLE]

To use the CWAE with the target distribution $P_{Z}$ being a mixture of Gaussians, it is sufficient to replace $d_{\mathrm{MMD}}$ in (6) by (V). We further take the logarithm of the squared Cramer-Wold distance to improve the balance between this term and the MSE [13, Footnote 2]. Taking all this together, we define the SeGMA objective function.

Definition 1**.**

Let $P_{Z}=\sum_{k=1}^{K}p_{k}N(\mu_{k},I)$ be a target distribution and let $\hat{X}=X\cup X_{L}$ be composed of unlabeled and labeled data, where $Y_{L}$ denotes labels of $X_{L}$. The aim of SeGMA is to find minimizers of the following optimization problem:

[TABLE]

where $\alpha,\beta$ are trade-off parameters and where $P_{Y|Z}$ and $d_{\gamma}^{2}$ are given in (7) and (V), respectively.

As it can be seen in this definition, we choose to model each class with a single Gaussian component with identity covariance matrix. One may argue that this choice is very restrictive, limiting the expressiveness of our model. In practice, however, encoder and decoder are implemented by deep neural networks. Thus, while multiple components with trainable variances per class may lead to a lower loss, our preliminary experiments indicate that the restrictions for $P_{Z}$ in Definition 1 are unproblematic from a practical perspective, given that the encoder and decoder networks have sufficient capacity. At the same time, the capacity of the encoder and decoder needs to be restricted to ensure that data points from the same class are mapped to the same component in latent space.

SeGMA is trained similar as CWAE, with a slight modification caused by the use of labeled data. Every mini-batch, which is composed of labeled and unlabeled data, is processed by the encoder and decoder. It is common practice that half of data in a mini-batch is labeled555If the number of labeled data is too low to achieve this, we can either decrease the number of samples in a mini-batch, or use oversampling techniques to virtually increase the number of labeled data. [11]. If we have additional knowledge about class proportions, we use these proportions for selecting labeled examples in each mini-batch. The cross entropy $\mathcal{L}$ is computed only for labeled examples, while $\mathrm{MSE}$ and $d_{\gamma}^{2}$ are evaluated for all data. During back-propagation, we update both weights of the encoder and decoder, as well as the means $\mu_{k}$ of Gaussians. Component masses $p_{k}$ are set to reflect the proportions of classes in the set of the known labels $Y_{k}$. While component masses can also be trained, our preliminary experiments show that this does not improve model performance significantly. Updating the means of Gaussians, however, allows us to better organize the class structure in the latent space. Namely, our model can place similar classes closer together, while significantly distinct ones can be put far apart. This property can be useful in analyzing the similarity between classes after successful training, which would not be possible using a separate discrete class variable.

VI Experiments

In this section, we present the results of our experiments. Our goal is to compare with similar methods on standard benchmark tasks as well as to present interesting properties of SeGMA that are difficult to obtain using existing approaches.

VI-A Experimental setting

For the evaluation, we chose three datasets of varying complexity, namely MNIST [30], SVHN [31] and CelebA [32], which are commonly used for comparing generative models [12, 2]. For these datasets we randomly chose, respectively, 100, 1000 and 1000 labeled samples, while the rest of the total 60000, 604388 and 182637 samples remained unlabeled. Since CelebA labels consist of multiple tags (i.e. sex, presence of eyeglasses, presence of facial hair, etc.), we kept only the two most balanced tags (’smiling’ and ’male’) and created four classes: ’not male, not smiling’, ’not male, smiling’, ’male, not smiling’ and ’male, smiling’.

We compare SeGMA to previous semi-supervised generative models based on VAEs: M1+M2 [11] and ADGM [12]. Additionally, we compare to Triple-GAN [33], which, however, does not allow for reconstructing examples. This rules out some of the experimental studies performed in this paper, e.g., interpolation, which explains why Triple-GAN does not appear in Sections VI-C and VI-D. These methods were reimplemented based on an open repository666https://github.com/wohlert/semi-supervised-pytorch, the original authors’ codes777https://github.com/dpkingma/nips14-ssl for M1+M2, https://github.com/larsmaaloee/auxiliary-deep-generative-models for ADGM and https://github.com/zhenxuan00/triple-gan for Triple-GAN. and the description of these models in the papers888We we unable to use original authors codes directly, because they are based on a version of Theano that is no longer supported.. We excluded methods that do not focus on generative properties (although they are based on generative models), e.g. [26], or which are not able to generate samples from a desired class, e.g. [4]. To the best of our knowledge, these are the only deep models, which introduce partial supervision to learn probability distributions of specific classes (not only the whole data density). For a comparison between classical WAE, VAE and GAN, we refer the reader to [13] and [2].

Contrary to the pipelines presented in [11, 12], we used no preprocessing on any of the datasets except for CelebA, where we followed [2] and cropped and then resized images to a resolution of 140$\times$140 and 64$\times$64, respectively. For experiments on MNIST and SVHN we kept the architectures from the original papers, i.e., fully connected architectures for M1+M2 and ADGM and convolutional networks in the case of Triple-GAN. For a fair comparison with VAE-based methods, SeGMA was also instantiated with fully connected networks – we used ReLU activation functions and layer sizes $1024-1024-10-1024-1024-784$ and $786-786-786-786-786-20-786-786-786-786-786-3072$, respectively (i.e., the latent space has dimension $D=10$ and $D=20$ for MNIST and SVHN). For CelebA, the architecture was based on the one from [13] and then adapted to the unsupervised setting; the details can be found in the Appendix. We initialize the network parameters using the Glorot initialization [34] and we initialize the means $\mu_{k}$ of the Gaussian components so that distance between each two means is $1$ at the start of the training. To show that SeGMA does not need much fine-tuning on the validation set (which is often impractical in semi-supervised settings with a small number of available labels), we used the same values of hyperparameters ($\alpha=5$, $\beta=10$, $\gamma=(\frac{4}{3N_{c}})^{\frac{2}{5}}$, where $N_{c}$ is the number of samples per class in a single batch, learning rate $=3\cdot 10^{-4}$) for all experiments and adapted only the network architecture to each data set. The reader is referred to Section VI-E, where the sensitivity to some of the parameters is analyzed. We do not use any regularizers (weight decay, dropout, batch normalization), since we want to show that our model works in the simplest possible setting. Additional regularization might further improve its performance.

VI-B Generative performance

First, examine the quality of generated samples and show that SeGMA is capable of generating representative samples of a given class. For this purpose, we generate samples from each Gaussian component and pass these through the decoder.

Randomly selected samples are presented in Figs. 3, 4 and 5 – each row in the images corresponds to samples related to a separate Gaussian component for SeGMA, and to samples determined by a separate discrete class variable in M1+M2, ADGM and Triple-GAN, respectively. It is evident that SeGMA generates realistically-looking samples. The produced samples are diverse and as sharp as samples returned by the corresponding CWAE model, see [13, Figure 2]. While ADGM generates samples of comparable quality for MNIST, the images produced for SVHN are less colorful. For CelebA, ADGM returns grainy images, which contain many anomalies. It may be caused by an encoder network, which has to combine convolutional layers applied to an image with dense layers applied to one-hot vector representing the class999The authors of ADGM did not examine their method in the case of convolutional neural networks.. M1+M2 gives reasonable results only for MNIST while for other data sets, the images are blurry and there is not enough variance between different images which might follow from the mode collapse. We wish to mention that the results reported in [11] displayed better quality for SVHN. We believe that this is due to specific preprocessing and careful hyperparameter selection, which is difficult to accomplish in a typical semi-supervised setting where the number of labeled data available for a validation stage is limited. Triple-GAN generates very realistically-looking and sharp samples, but at the same time produces anomalies (e.g. the third image in the seventh row of Fig. 4a). We believe that the quality of Triple-GAN samples comes at the cost of extensive hyperparameter tuning and architecture search. In particular, we tried to use the architecture for Cifar-10 from the original work in order to run the method on CelebA dataset, but the method was not able to generate faces even after few hundred steps of training. For MNIST and SVHN, we used the architectures and hyperparameters from [33]; these were originally tuned on a held-out validation set with all labels [33], which is an impractical approach to semi-supervised learning [35]. In Section VI-E we show that such precise choice of hyperparameters is not required for SeGMA.

To provide an additional quantitative assessment of sample quality, we calculate the Fréchet Inception Distance (FID) [36], which is a standard score for generative models. To calculate FIDs, we used 10000 images sampled from the target distribution in the latent space. To better understand the connection between the generativity, semi-supervision, and a mixture of Gaussians as a target distribution, we also show FIDs for an unsupervised variant of SeGMA (i.e. for $\beta=0$) and for a CWAE with a single spherical Gaussian as a target distribution (i.e. for $\beta=0$ and $K=1$). Moreover, to show the dependence of SeGMA on the type of architecture, for SVHN we also report FIDs for a convolutional neural network. It is evident from Table I that the FIDs of SeGMA and ADGM are comparable across datasets, which is slightly surprising when one looks at the samples generated by these models. The use of a convolutional architecture allows to significantly improve the performance of SeGMA on SVHN. It is interesting that the FID of CWAE is better than both versions of SeGMA on SVHN. It suggests that deep generative models can be trained more easily on unimodal target distributions than on a more complex mixture of Gaussians. In this case, we need to pay an additional cost (i.e., slightly lower quality of samples) for building a generative model for each class. Nevertheless, this observation does not hold for CelebA and MNIST, where the information of class labels allows to build better generative models for the entire data.

Visual inspection of the results suggests that the components in each model correspond to the correct classes (examples in each row of the figures have the same class). There are only few errors (e.g., the digits "2" and "3" in the second and sixth row of Fig. 3d or the digit "5" in the ninth row of Fig. 4d). Notably, similar errors occur also in M1+M2 (e.g., digit "6" in the first row of Fig. 4b) and ADGM (e.g., digit "8" in the last row of Fig. 4c) despite the fact that these methods require explicit discrete latent variables. To get deeper insight into this aspect, we measured the classification accuracy on test sets. More precisely, we encoded each test image using an encoder network and classify it using (7) in the latent space. Table II shows that SeGMA gives better results than competitive methods on harder datasets such as SVHN and CelebA, especially when using a convolutional architecture. This is consistent with our previous observation that SeGMA obtained higher quality of samples on these sets.

VI-C Interpolation

ADGM and M1+M2 use a single Gaussian as a target distribution for the style of all classes, but add an additional discrete distribution to identify the class. By one-hot encoding this discrete class variable, interpolation between classes is possible in principle; e.g., by changing the class indicator from $(1,0,0,\dots)$ to $(0,1,0,\dots)$ via $(0.5,0.5,0,\dots)$. However, since $(0.5,0.5,0,\dots)$ has never appeared during training, these methods produce realistically-looking interpolations between samples in the same class, but have problems with smooth interpolation between samples in different classes. In contrast, SeGMA uses a single continuous latent space in which the discrete class label is implicit in the identity of the mode of a potentially multimodal target distribution (e.g., Gaussian component in the case of a mixture of Gaussians). In other words, the samples from all classes are placed in the same continuous space. Ensuring that the different mixture components are not too far apart – thus ensuring that also the space between component means is populated by training data – allows us to obtain realistic interpolations.

To confirm this claim experimentally, we randomly pick a pair of images and map them to the latent space. Next, we generate linear interpolations between these encoded points and decode these interpolations using a decoder network. To obtain analogical effect for ADGM, we interpolate both the latent codes and the one-hot representations of classes.

The results of ADGM and SeGMA are presented in the Fig. 6f (we omit M1+M2 interpolations because of their poor quality). It can be seen that interpolations produced by SeGMA have better quality than the ones obtained using ADGM. It is evident that many interpolations generated by ADGM are outside of the true data density (e.g. images in the sixth column of the first, second, fourth and fifth rows in the case of MNIST). For SVHN and CelebA, interpolations are often blurry or unrealistic (second row of CelebA and fifth row for SVHN). Note that the interpolations produced by SeGMA can also pass through the regions of low density, which lie between Gaussian components (e.g. fourth row in the case of MNIST). This negative effect could be however partially fixed using more sophisticated interpolations than the linear one [37, 38], which is not fully justified for this type of distribution. Nevertheless, we can observe very interesting geometrical properties of the class structure obtained by SeGMA: The interpolation between "6" and "9" (first row) leads to generating a digit "4" (first row). This suggests that the class of digits "4" is localized between classes of digits "6" and "9". Similar effect can be observed for the second row, where the model passes through the class of digits "3".

We support above claims by calculating FID scores for images generated from interpolations. Specifically, we took two random images from the test data, encoded them into the latent space, randomly picked a point on the line connecting the encoded images and decoded it. Table III shows the FIDs for 10000 such decoded samples when compared against the real data. It is evident that FIDs of SeGMA are significantly better than the ones obtained by competitive methods.

VI-D Style Transfer and Class Intensity

Semi-supervised generative models based on the VAE generate images by decoding a data point sampled from the target distribution and a one-hot vector describing the class label. If we flip the class vector, but keep the latent code unchanged, then the model returns an image with similar style, but from a different class. In other words, these models disentangle class and style information in latent space. In this experiment, we investigate whether a comparable style transfer can be achieved for our model.

In the case of SeGMA, we cannot simply change the class label, because we do not have an individual vector describing the class. However, we can move a latent code generated by one Gaussian component to the corresponding location w.r.t. the mode of another Gaussian component. In other words, we move a latent code $z$ by a vector between the mean of target Gaussian $N(\mu_{t},I)$ and the mean of the initial Gaussian $N(\mu_{s},I)$ from which it was generated, i.e.:

[TABLE]

In addition, we can interpolate between $z$ and $\hat{z}$ to change the class label gradually. Moving along this line in latent space is more natural than moving along a corresponding line between two one-hot encoded class vectors, as it would be necessary for semi-supervised VAE models. In the continuous latent space, it is natural to assume that close points correspond to similar images; for one-hot encoded class vectors or a discrete latent space, notions of closeness and similarity are inadequate.

Based on the above discussion, we encode an image into the latent space and gradually change its class label in a direction to the randomly selected class according to (10). Sample results of this operation for SeGMA and analogous results for ADGM are presented in Fig. 6l. Although the style transfer leads to different images for SeGMA and for ADGM (see right-most images in Fig. 6l), it can be seen that the style is indeed maintained for SeGMA. In addition to that, the images obtained via interpolation are more natural for SeGMA. As mentioned, this is because ADGM encodes only discrete class labels while SeGMA treats it as continuous variable.

An interesting property of our model is that we can not only change the class of the presented data point, but also increase or decrease the intensity of certain stylistic aspects. We can achieve this property by moving the latent representation $z$ of a data point away from a class which lacks the considered stylistic aspect. More precisely, let $z$ be a latent code of an image that belongs to the initial Gaussian $N(\mu_{s},I)$. We can move $z$ away from an anti-target Gaussian $N(\mu_{t},I)$ by setting $\hat{z}=z+\alpha(\mu_{s}-\mu_{t})$ for some $\alpha>0$. For example, by using an image of a male face and then moving it away from the Gaussian representing the ‘not male‘ class, we can add more masculine features to the picture. Examples of this operation are presented in Fig. 6 for $\alpha=0.5$. In the first three rows, the samples are moving away from the "not male/smiling" class, while in the fourth row, it is moving away from the "male/smiling" class. It can be observed that the man in the third row gains more masculine feature, while the woman in the fourth row gains more feminine features. This suggests that the presence of a beard is recognized as a typical male attribute and was encoded far from Gaussian components connected with female images.

VI-E Sensitivity Analyses

We finally analyze the sensitivity of SeGMA to its parameters. For the sake of brevity we only present results for the fully connected architecture trained on SVHN.

First, we examine the sensitivity to the selection of hyperparameters. Fig. 7 shows test accuracy and FIDs obtained for different values of $\alpha$ (parameter for Cramer-Wold loss) and $\beta$ (parameter for cross-entropy loss). Visual inspection suggests that the loss component representing the reconstruction error is crucial for both classification and generativity. The reconstruction error component has no hyperparameter, i.e., its relative weight increases if $\alpha$ and $\beta$ are both small. Indeed, increasing either $\alpha$ or $\beta$ without increasing the capacity of the model (number of layers and their width, etc) increases FIDs and decreases test accuracies, because the network no longer tries to distinguish important details of the examples. While $\beta$ is positively correlated with test accuracy its influence on FIDs seems weak. More generally, judging from the range of accuracies and FIDs in Fig. 7 (most accuracies and FIDs range between 65% and 74% and between 54 and 62, respectively), we conclude that SeGMA is comparably stable w.r.t. the hyperparameters, which obviates the need for fine-tuning.

We next analyze the influence of the number of latent dimensions on the performance. The results presented in Fig. 8 (left) demonstrate that a too small latent space creates a bottleneck that prevents the model from separating examples from different classes and from generating high-quality samples. Introducing too many dimensions in the latent space leads to a small but noticeable decrease of performance. In summary, however, the sensitivity of SeGMA on the number of latent dimensions is weak.

Finally, we examine the dependence on the number of labels available in the training set. As expected, Fig. 8 (right) shows that a higher number of labeled data leads to better classification accuracy. In contrast, increasing the number of labeled samples seems to have a small negative effect on the quality of generated samples (measured by FID scores). As the network has to fit more labeled data, a smaller portion of its capacity can be used to reduce other loss components. This shows that generative models are capable of producing realistic images without any information from class labels. This is consistent with the results in Table I, where unsupervised SeGMA achieved better FIDs than its supervised version.

VII Conclusion

We constructed SeGMA, a semi-supervised generative model, which combines a WAE-MMD framework with a mixture of Gaussians as a target distribution. By connecting each component with a specific class, SeGMA is capable of generating class representatives of high quality. Since a discrete class label is implicitly encoded in the same latent space as the style variable, we can interpolate between samples from different classes, which was not possible in existing semi-supervised generative models. In particular, we can change the class labels gradually while preserving the style of a given sample. Moreover, SeGMA allows for increasing (decreasing) the influence of a given class by moving this example in the (opposite) direction of the corresponding Gaussian component. For example, moving a sample away from the “female” components in latent space resulted in growing a beard in male celebrities.

Due to the use of a simple auto-encoder architecture from a classical WAE model, SeGMA scales better with the complexity of the dataset than other semi-supervised generative approaches. In particular, SeGMA requires fewer dimensions in the latent space (20 dimensions for SVHN) than ADGM (300 dimensions for SVHN), which is composed on six neural networks. Making use of Cramer-Wold distance as MMD penalty, we do not need to sample during training, which saves computational resources and increases model stability as compared to, e.g. ADGM which requires sampling. Finally, SeGMA can be more easily adapted to different (e.g. convolutional) network architectures than the previous semi-supervised models.

Appendix A Architectural Details for Experiments with CelebA

For SeGMA, the encoder $\mathcal{E}$ consists of two convolutional layers with 32 filters, two convolutional layers with 64 filters, and three fully connected layers of size $786-786-32$, where $D=32$ is the latent space dimension. The decoder $\mathcal{D}$ consists of three fully connected layers of size $786-786-1024$, one transposed convolutional layer with 64 filters, two transposed convolutional layers with 32 filters, and a final transposed convolutional layer with 3 filters.

For M1, the encoder $\mathcal{E}$ consists of two convolutional layers with 32 filters, two convolutional layers with 64 filters, and two fully connected layers of size $300-50$, where $D=50$ is the latent space dimension. The decoder $\mathcal{D}$ consists of two fully connected layers of size $300-1024$, one transposed convolutional layer with 64 filters, two transposed convolutional layers with 32 filters, and a final transposed convolutional layer with 3 filters. The latent space has dimension $D=300$. For M2, encoder and decoder are fully connected with sizes $300$ and $300-300$, respectively.

For ADGM, the encoder $\mathcal{E}$ consists of two convolutional layers with 32 filters and two convolutional layers with 64 filters. The result is a flattened vector. After that every edge in the probabilistic graph of this model is implemented by a neural network with two fully connected layers of size $500-500$, with the latent space dimension $D=300$. Also the decoder $\mathcal{D}$ implements every edge in the probabilistic graph by a neural network with two fully connected layers of size $500-500$. The resulting image is further processed by a transposed convolutional layer with 64 filters, two transposed convolutional layer with 32 filters, and a final transposed convolutional layer with 3 filters.

All networks are implemented using ReLU activation functions and all convolution filters are of size 4x4 and stride 2x2.