Disentangling and Learning Robust Representations with Natural Clustering

TL;DR

This paper introduces N-VAE, a model that disentangles class-specific and shared factors of variation in data, improving generative capabilities and class-dependent factor detection.

Contribution

The paper proposes N-VAE, a novel model with a class-conditioned and shared latent space for disentangling multimodal generative factors.

Findings

Effective separation of class-specific and shared factors.

Ability to generate novel samples with unseen characteristics.

Improved detection of class-dependent generative factors.

Abstract

Learning representations that disentangle the underlying factors of variability in data is an intuitive way to achieve generalization in deep models. In this work, we address the scenario where generative factors present a multimodal distribution due to the existence of class distinction in the data. We propose N-VAE, a model which is capable of separating factors of variation which are exclusive to certain classes from factors that are shared among classes. This model implements an explicitly compositional latent variable structure by defining a class-conditioned latent space and a shared latent space. We show its usefulness for detecting and disentangling class-dependent generative factors as well as its capacity to generate artificial samples which contain characteristics unseen in the training data.

Figures

Equations

$$\mathcal{L}(\mathbf{x})=-\underset{\text{Regularization cost}}{\underbrace{D_{KL}(q(\mathbf{z}|\mathbf{x})\Vert p(\mathbf{z}))}}+\underset{\text{Reconstruction cost}}{\underbrace{E_{q(\mathbf{z}|\mathbf{x})}[\log p(\mathbf{x}|\mathbf{z})]}}$$

$$\log p(\mathbf{x},y)\ge \mathcal{L}(\mathbf{x},y)$$

$$+\log p(\mathbf{x},y,\mathbf{z},\mathbf{\pi })]$$

$$\mathcal{L}(\mathbf{x},y)=E_{q(\mathbf{z}|\mathbf{x})}[\log p(\mathbf{x}|y,\mathbf{z})]-D_{KL}(q(\mathbf{z}|\mathbf{x})\Vert p(\mathbf{z}))$$

$$-D_{KL}(q(\mathbf{\pi }|\mathbf{x})\Vert p(\mathbf{\pi }|y))+\log p(y)$$

$$q_{\varphi }(\mathbf{\pi }|\mathbf{x})=Dir(\mathbf{\pi };{\alpha }_q\cdot \alpha (\mathbf{x};\varphi ))$$

$$q_{\varphi }(\mathbf{z}|\mathbf{x})=\mathcal{N}(\mathbf{z};\mathbf{\mu }(\mathbf{x};\varphi ),\mathbf{\sigma }(\mathbf{x};\varphi )\cdot \mathbf{I})$$

$${\mathcal{L}}_{obj}(\mathbf{x},y)$$

$$-D_{KL}(q_{\varphi }(\mathbf{z}|\mathbf{x})\Vert p(\mathbf{z}))+\log (q_{\varphi }(y|\mathbf{x}))$$

$${\mathcal{L}}_{{\beta }_c}$$

$$-{\beta }_c{D}_{KL}(q_{\varphi }({\mathbf{z}}_c|\mathbf{x})\Vert p(\mathbf{z}))+\log (q_{\varphi }(y|\mathbf{x}))$$

$$D_{KL}(q(\mathbf{z},\mathbf{\pi }|\mathbf{x},y)\Vert p(\mathbf{z},\mathbf{\pi }|\mathbf{x},y))$$

$$=E_{q(\mathbf{z},\mathbf{\pi }|\mathbf{x},y)}[\log q(\mathbf{z},\mathbf{\pi }|\mathbf{x},y)-\log p(\mathbf{z},\mathbf{\pi }|\mathbf{x},y)]$$

$$=E_{q(\mathbf{z},\mathbf{\pi }|\mathbf{x},y)}[\log q(\mathbf{z},\mathbf{\pi }|\mathbf{x},y)$$

$$-\log p(\mathbf{x},y,\mathbf{z},\mathbf{\pi })]+\log p(\mathbf{x},y)$$

$$=-\mathcal{L}(\mathbf{x},y)+\log p(\mathbf{x},y)$$

$$\log p(\mathbf{x},y)\ge \mathcal{L}(\mathbf{x},y)$$

$$=E_{q(\mathbf{z},\mathbf{\pi }|\mathbf{x},y)}[-\log q(\mathbf{z},\mathbf{\pi }|\mathbf{x},y)+\log p(\mathbf{x},y,\mathbf{z},\mathbf{\pi })]$$

$$=E_{q(\mathbf{z},\mathbf{\pi }|\mathbf{x},y)}[-\log q(\mathbf{z},\mathbf{\pi }|\mathbf{x},y)$$

$$+\log (p(\mathbf{x}|y,\mathbf{\pi },\mathbf{z})p(\mathbf{z})p(\mathbf{\pi }|y)p(y))]$$

$$=E_{q(\mathbf{z},\mathbf{\pi }|\mathbf{x},y)}[\log p(\mathbf{x}|y,\mathbf{\pi },\mathbf{z})]$$

$$-E_{q(\mathbf{z},\mathbf{\pi }|\mathbf{x},y)}[\log q(\mathbf{z}|\mathbf{x},y)-\log p(\mathbf{z})]$$

$$-E_{q(\mathbf{z},\mathbf{\pi }|\mathbf{x},y)}[\log q(\mathbf{\pi }|\mathbf{x},y)-\log p(\mathbf{\pi }|y)]$$

$$+E_{q(\mathbf{z},\mathbf{\pi }|\mathbf{x},y)}[\log p(y)]$$

$$=E_{q(\mathbf{z}|\mathbf{x})}[\log p(\mathbf{x}|y,\mathbf{z})]-D_{KL}(q(\mathbf{z}|\mathbf{x})\Vert p(\mathbf{z}))$$

$$-D_{KL}(q(\mathbf{\pi }|\mathbf{x})\Vert p(\mathbf{\pi }|y))+\log p(y)$$

Full text

I Introduction

Disentangled representations, defined as those where each latent variable is responsible for only one generative factor in the data while being relatively invariant to other factors [1], allow for more easily interpretable models, [2], and facilitate generalization to previously unseen combinations of features, [3]. They also generalize to new domains that share underlying factors of variation, [4].

Typically, variational autoencoder (VAE) [5] based disentanglement methods impose an isotropic Gaussian prior on their latent space. However, any dataset that can be separated into classes will present some form of multimodality. Despite the unimodal prior, in these scenarios, the inferred posterior is often multimodal. This creates dependencies between latent dimensions and reduces reconstruction accuracy, [6].

Moreover, there can exist underlying factors of variation in data that are exclusive to certain classes. An example of this phenomenon is found in the MNIST dataset. The number seven can be drawn with a horizontal line crossing its stem or without it. The number two can have a flat base or one with a loop. We show how these class-exclusive factors are disentangled by our proposed model in fig. 1. Other factors of variability, such as stroke thickness or azimuth, are shared among all classes, as can be seen in fig. 2.

The natural clustering prior [1] states that probability mass concentrates on a set of low dimensional manifolds. Each manifold is associated with a possible outcome of categorically distributed phenomena in the data, such as class membership. Our model, natural clustering VAE (N-VAE), conditions some of its latent variables on samples from a categorical distribution. By incorporating inductive biases about compositional structure in the data, it is able to capture class-dependent factors of variation which are usually ignored by other generative modelling techniques.

Our main contributions are the following: 1) We formulate a lower bound on the joint log-likelihood of inputs and class labels. It can be optimized approximately by simply adding a classification objective to the standard VAE cost function. 2) We propose N-VAE, a model that is capable of capturing both generative factors which are class-dependent and ones that are shared among classes. To the best of our knowledge, we are the first to disentangle these two types of factors. 3) We show how N-VAE can be used for detection and disentanglement of mode-dependent factors of variation. We also evaluate the expressivity111We understand expressivity as the number of possible input configurations that are explained by a model, [1]. of N-VAE relative to other class-conditioned generative models by comparing the usefulness of their generated samples for dataset augmentation.

II Preliminaries

We first give a brief overview of the variational autoencoder, as N-VAE builds upon this model. We then give a recap of some recent techniques for disentangling in VAEs. We show that, despite the popularity of these models, they are not well equipped to disentangle multimodally distributed generative factors.

II-A The Variational Autoencoder

The VAE approximates the intractable posterior over a set of latent variables $p(\mathbf{z}|\mathbf{x})$ with an approximate distribution $q(\mathbf{z}|\mathbf{x})$. $D_{KL}(q(\mathbf{z}|\mathbf{x})\,\|\,p(\mathbf{z}|\mathbf{x}))$ is minimized by optimizing the Evidence Lower Bound (ELBO) $\mathcal{L}(\mathbf{x})\leq\log{p(\mathbf{x})}$:

[TABLE]

$\mathcal{L}(\mathbf{x})$ becomes equal to $\log{p(\mathbf{x})}$ when $q(\mathbf{z}|\mathbf{x})\!=\!p(\mathbf{z}|\mathbf{x})$. The VAE framework parametrises both $p(\mathbf{x}|\mathbf{z})$ and $q(\mathbf{z}|\mathbf{x})$ with neural networks. Optimization is performed through gradient descent, using the reparameterisation trick, [5, 7].

The encoder, $q_{\phi}(\mathbf{z}|\mathbf{x})$, is an approximate inference model which parametrises a distribution over the latent variables $\mathbf{z}$. A Gaussian with diagonal covariance is usually selected: $\mathcal{N}(\mathbf{z};\boldsymbol{\mu}(\mathbf{x};\phi),\boldsymbol{\sigma}(\mathbf{x};\phi)\!\cdot\!\mathbf{I})$. The decoder, $p_{\theta}(\mathbf{x}|\mathbf{z})$, is a generative model that reconstructs the input $\mathbf{x}$ from the latent variables $\mathbf{z}$. The prior over the latent variables is usually chosen to be $p(\mathbf{z})\!=\!\mathcal{N}(\mathbf{z};\,\mathbf{0},\mathbf{I})$.

II-B Regularization-based approaches to disentangled VAEs

The authors of [8] and [9] propose $\beta$-VAE. This model reweights the ELBO by multiplying the regularization cost by $\beta>1$. This pushes the approximate posterior to be closer to the factorial prior. In [10, 11, 12], the ELBO’s regularization cost is decomposed into the mutual information between inputs and latent variables and the total correlation of the latent space, [13]. The authors propose different modifications to the regularization term of the VAE objective, all of which aim to minimize the total correlation without decreasing the mutual information. In [14], an approach that penalizes the L2 distance between the second order moments of the aggregate approximate posterior and factorial prior is proposed.

II-C The problem with disentangling multimodal factors

In fig. 3, we show that the distribution of latent variables inferred by $\beta$-VAE [8] on the MNIST test set is multimodal. Roughly, one mode belongs to each digit. When traversing any given dimension of the latent space, we are likely to pass through low probability regions, generating poor quality images, and eventually switch digits.

In models with unimodal priors, such as the ones proposed in [8, 9, 10, 11, 12, 14], multimodally distributed explanatory factors tend to occupy multiple dimensions of the latent space, becoming entangled with other attributes. We posit that expressing mode-dependent generative factors, such as the ones shown in fig. 1, can result in a multimodal approximate posterior and thus, a high cost under the ELBO’s KL term. These factors are often only learned if their impact on the reconstruction accuracy is significant enough to mitigate the large KL cost.

III Proposed Method

Let $\{\mathbf{x}^{(i)},y^{(i)}\}^{N}_{i=1}$ be a dataset consisting of $N$ i.i.d observations of some variable $\mathbf{x}^{(i)}\,{\in}\,\mathbb{R}^{D}$ and its class label be $y^{(i)}\in\{1,\,...,\,L\}$. We propose a generative model which describes $\mathbf{x}^{(i)}$ as being generated by a set of latent variables $\mathbf{z}^{(i)}$ and a class label $y^{(i)}$. The class probabilities are given by $\boldsymbol{\pi}^{(i)}$. Unlike [15], we treat $\boldsymbol{\pi}^{(i)}$ as a continuous latent variable over which we perform inference. We will omit the superscript (i) in order to maintain notation uncluttered in the rest of this paper. The joint distribution of our variables can be factorized as: $p(\mathbf{x},y,\mathbf{z},\boldsymbol{\pi})=p(\mathbf{x}|y,\mathbf{z})p(y|\boldsymbol{\pi})p(\boldsymbol{\pi})p(\mathbf{z})$.

III-A A Lower Bound on the Joint Log-Likelihood

We formulate a variational lower bound on $\log p(\mathbf{x},y)$:

[TABLE]

This expression can be expanded and rewritten as eq. 3. See appendix A. for a complete derivation.

[TABLE]

$\operatorname{\mathbb{E}}_{q(\mathbf{z}|\mathbf{x})}[\log p(\mathbf{x}|y,\mathbf{z})]$ is a reconstruction objective for our data vector $\mathbf{x}$ given $\mathbf{z}$ and $y$. $D_{KL}(q(\boldsymbol{\pi}|\mathbf{x})\,\|\,p(\boldsymbol{\pi}|y))$ penalizes the approximate distribution of $\boldsymbol{\pi}$ given $\mathbf{x}$ for differing from the true posterior of $\boldsymbol{\pi}$ given the class label $y$. This item can be interpreted as a classification objective. Its value will go to [math] as $q(\boldsymbol{\pi}|\mathbf{x})$ approaches a one-hot vector with $\pi_{y}\!=\!1$. It is the most important difference between our lower bound and that of a standard VAE. $\log p(y)$ is constant with respect to our model’s parameters.

III-B Proposed Model: N-VAE

We place a zero mean isotropic unit variance Gaussian prior over $\mathbf{z}$: $p(\mathbf{z})=\mathcal{N(\mathbf{z};\,\mathbf{0},\mathbf{I})}$. We model the conditional over class labels with a categorical distribution: $p(y|\boldsymbol{\pi})=Cat(y;\,\boldsymbol{\pi})$. Its parameters, $\boldsymbol{\pi}$, follow a symmetric Dirichlet distribution, as we assume all classes to be equally probable: $p(\boldsymbol{\pi})=Dir(\boldsymbol{\pi};\,\alpha_{p}\!\cdot\!\mathbf{1}_{L})$. As the Dirichlet distribution is conjugate to the categorical, the posterior will also be a Dirichlet distribution: $p(\boldsymbol{\pi}|y)=Dir(\boldsymbol{\pi};\,\alpha_{p}\!\cdot\!\mathbf{1}_{L}+\mathbf{c}_{y})$. Note that $\mathbf{c}_{y}$ is a one-hot representation of the class label $y$.

We use an isotropic Gaussian distribution for $q(\mathbf{z}|\mathbf{x})$ and a Dirichlet for $q(\boldsymbol{\pi}|\mathbf{x})$. The neural networks that parametrise both of these approximate posteriors share parameters $\phi$.

[TABLE]

We define $\mathbf{z}$ as the concatenation of two vectors: $\mathbf{z}=[\mathbf{z}_{c}^{\intercal},\mathbf{z}_{s}^{\intercal}]^{\intercal}$. The class-dependent variables, $\mathbf{z}_{c}$, are the concatenation of a set of latent vectors, each of which is associated with a different class: $\mathbf{z}_{c}=[\mathbf{z}_{c1}^{\intercal},\mathbf{z}_{c2}^{\intercal},\,...,\,\mathbf{z}_{cL}^{\intercal}]^{\intercal}$. We want $\mathbf{z}_{c}$ to explain intraclass factors of variation. We refer to $\mathbf{z}_{s}$ as shared latent variables. They will express generative factors that are common to multiple classes. This allows for an efficient use of training samples, as these factors do not need to be learned separately for each input mode.

For a given class $y$, we compute the variable $\mathbf{z}_{cy}=vec(\mathbf{c}_{y}\odot[\mathbf{1}_{L},[\mathbf{z}_{c1},\mathbf{z}_{c2},\,...,\,\mathbf{z}_{cL}]^{\intercal}])$.222We refer to the element-wise product with broadcasting by $\odot$. $vec(\cdot)$ denotes the vectorization of a matrix into a column vector. The input, $\mathbf{x}$, is regenerated from $[\mathbf{z}_{cy}^{\intercal},\mathbf{z}_{s}^{\intercal}]^{\intercal}$. The one-hot class label masks out the class-dependent latent variables for all non-relevant classes. It also introduces a class-dependent bias in the decoder’s first activation layer when $\mathbf{c}_{y}$ is multiplied with its corresponding weight vector. The effects of this term are discussed in appendix B. The decoder is parametrized by $\theta$, taking the form $p_{\theta}(\mathbf{x}|y,\mathbf{z})\!=\!f(\mathbf{x};y,\mathbf{z},\mathbf{\theta})$. The complete graphical model is displayed in fig. 4. N-VAE is closely related to [16], where domain dependent, class dependent and shared information are captured in separate latent spaces.

III-C Resulting objective function and optimization

N-VAE’s encoder is both a feature extractor and a classifier. For two Dirichlet distributions, $D_{KL}(q_{\phi}(\boldsymbol{\pi}|\mathbf{x})\,\|\,p(\boldsymbol{\pi}|y))$ can be upper bounded by the inferred class label’s negative log likelihood: $-\log(q_{\phi}(y|\mathbf{x}))+k$, with $k$ being a constant. Using this approximation and dropping all constants with respect to model parameters, we optimize eq. 3 as:

[TABLE]

The KL divergence between the approximate posterior over $\mathbf{z}$ and its prior distribution is calculated for all dimensions of $\mathbf{z}\!=\![\mathbf{z}_{c}^{\intercal},\mathbf{z}_{s}^{\intercal}]^{\intercal}$. However, not all dimensions of $\mathbf{z}_{c}$ are used to reconstruct every input. Most of them are masked out. The encoder learns to match the prior distribution for all dimensions of $\mathbf{z}_{c}$ which are not assigned to the correct class, making them non-informative. In other words, the encoder learns to model class-specific factors of variation.

IV Experiments

We consider three datasets which are separated into discrete classes: MNIST, Fashion-MNIST, and the cropped Extended Yale Face Dataset B. We preprocess the latter by only keeping images with illumination angles smaller than 100°and scaling images to $64{\times}64$, [17].

The training objective is reweighed in order to discourage $\mathbf{z}_{c}$ from learning factors of variation that are common to multiple classes:

[TABLE]

We set $\beta_{c}=2$. However, we find that N-VAE is relatively insensitive to the value of this parameter as long as $\beta_{c}>1$.

IV-A Detecting class-dependent factors of variation

In MNIST, there exist both class-exclusive and shared generative factors. However, in Fashion-MNIST, most of the continuous factors of variation seem to be intra-class. Classes such as sandals and t-shirts only share the factor of colour intensity. In Yale Ext B, the only continuous factors of variation are the elevation and azimuth of image illumination.

N-VAE’s KL divergence values for $\mathbf{z}_{s}$ and $\mathbf{z}_{c}$ act as detectors for class-dependent and shared information, fig. 5. The class-dependent variables carry the most information for Fashion-MNIST. For Yale Ext B, they are practically uninformative.

IV-B Disentangling generative factors

Latent traversals for Fashion-MNIST’s class-dependent variables are shown in fig. 6. In fig. 7, we show shared latent space traversals for the same dataset. The only generative factors which are shared among all classes seem to be color intensity and object width. In fig. 8, we show shared latent space traversals for Yale Ext B. We obtain all latent traversals by sampling values of $\mathbf{z}_{s}$ uniformly from the range $(-3,3)$.

We do not provide class-dependent latent space traversals for Yale Ext B, as $\mathbf{z}_{c}$ learns to be uninformative for this dataset. This is reflected in the values of $D_{KL}(q_{\phi}(\mathbf{z}_{c}|\mathbf{x})\,\|\,\ p(\mathbf{z}))$ from fig. 5. Changing $\mathbf{z}_{c}$ does not affect reconstructions.

By explicitly modelling multimodally distributed generative factors through class-dependent latent variables, the shared latent space is allowed to better match the factorial prior. Consequently, N-VAE is able to learn disentangled representations without elaborate modifications to the ELBO or extensive hyerparameter tuning.

IV-C Augmenting Datasets with Generated Samples

We train classifiers with different proportions of original and artificial (generated) samples and evaluate them using the original test sets. With this method, we not only test the discriminability of our generations but also how much of the original datasets’ variability they capture. A more diverse set of generated images will better constrain the classifiers’ parameters, resulting in higher accuracy.

We compare N-VAE with the conditional VAE (CVAE), [18], the conditional GAN (CGAN), [19] and the auxiliary classifier GAN (AC-GAN), [20]. All of these models use class labels during training and allow for class-conditional sample generation. Latent variables are sampled from $\mathcal{N(\mathbf{z};\,\mathbf{0},\sigma\!\cdot\!\mathbf{I})}$. A larger $\sigma$ allows sampling from regions of latent space which are less likely under the trainset. A large enough value should allow for the extrapolation of learned generative factors, producing more extreme characteristics or new combinations of features. Our intention is to show that generative models generalize better when an appropriate latent space structure is used, not to propose a new data augmentation scheme.

On MNIST, including artificial samples has a regularization effect for most generative models as is shown in fig. 9 a); Seeing more plausible data makes it more difficult for the classifiers to overfit to specific training samples. However, N-VAE is the only model for which there is a consistent decrease in classification error as $\sigma$ increases. It is able to produce feature configurations that complement those of the original training set. Fashion-MNIST is a more complex dataset which is harder to explain for generative models. In plot b), we see that using samples generated with N-VAE decreases the baseline classifier’s performance the least on this dataset.

We observe a significant performance decrease when training classifiers exclusively on artificial data in plots c) and d) from fig. 9. On MNIST, the conditional GAN produces the most useful samples for $\sigma=1$, N-VAE is a close second. On Fashion-MNIST, N-VAE provides the best results. In all scenarios, N-VAE performs best as $\sigma$ increases.

V Conclusion

N-VAE explains data in terms of shared latent variables and class-dependent latent variables. We have shown the utility of this type of model for identifying and disentangling class-dependent generative factors in data. In our data augmentation experiment, N-VAE is the only model that provides more informative samples for classification as $\sigma$ increases. This suggests that N-VAE is able to produce samples with new combinations of features, showing its capacity for generalization.

Acknowledgment

This work has been supported by the Spanish Ministry of Economy and Competitiveness and the European Social Fund through the project TIN2017-85854-C4-1-R, by the Government of Aragon (Reference Group T36_17R) and co-financed with Feder 2014-2020 ”Building Europe from Aragon.”

A: Derivation of eq. 2 and eq. 3

First, we show that our objective is a lower bound on the joint log-likelihood $\log p(\mathbf{x},y)$. Our starting point is the KL divergence between the true and approximate posteriors over $\mathbf{z}$ and $\boldsymbol{\pi}$, as this is the quantity which we wish to minimise.

[TABLE]

Using the non-negativity of the KL divergence, we can write: $\mathcal{L}(\mathbf{x},y)\leq\log p(\mathbf{x},y)$.

We make the following modelling choices:

• •

$\mathbf{x}$ is conditionally independent of $\boldsymbol{\pi}$ given $y$; $p(\mathbf{x}|\mathbf{z},y,\boldsymbol{\pi})=p(\mathbf{x}|\mathbf{z},y)$.

• •

Our inference model $q$ only takes $\mathbf{x}$ as an input. Therefore, $q(\boldsymbol{\pi}|\mathbf{x},y)=q(\mathbf{z}|\mathbf{x})$ and $q(\boldsymbol{\pi}|\mathbf{x},y)=q(\boldsymbol{\pi}|\mathbf{x})$.

We can rewrite eq. 2 in the following manner:

[TABLE]

B: The bias term in the decoder’s input

$\mathbf{z}_{cy}$ is obtained as $vec(\mathbf{c}_{y}\odot[\mathbf{1}_{L},[\mathbf{z}_{c1},\mathbf{z}_{c2},\,...,\,\mathbf{z}_{cL}]^{\intercal}])$. In this expression, we can see that $\mathbf{z}_{cy}$ is composed of the latent vectors predicted by the encoder, of which all but one is masked out, and a one-hot vector which results from $\mathbf{c}_{y}\odot\mathbf{1}_{L}=\mathbf{c}_{y}$. The latter explicitly informs the decoder about the class identity of the sample to be reconstructed. When multiplied with the corresponding weight layer, the one-hot term adds a class-dependent bias term $b_{y}$ to the decoder’s first activation layer.

The most likely value under the prior $p(\mathbf{z}_{c})=\mathcal{N}(\mathbf{z}_{c};\,\mathbf{0},\mathbf{I})$ is [math]. However, when we mask out latent variables, we set most of them to this value. This makes masked latent variables indistinguishable from samples from $p(\mathbf{z}_{c})$. Without $b_{y}$, the encoder is forced to predict distributions which place a lot of mass on large values of $\mathbf{z_{c}}$ in order to communicate the class of $\mathbf{x}$ to the decoder. In fig. 10, we show how the KL divergence grows for class-dependent latent variables when we remove $\mathbf{c}_{y}\odot\mathbf{1}_{L}$ and thus, the bias, from $\mathbf{z}_{cy}$. The explicit encoding of discrete information allows us to describe the data’s class-dependent generative factors with distributions that more closely resemble the prior. It allows for smooth interpolation of representations in the class-conditioned latent space around zero.

C: N-VAE samples with $\sigma=1.4$