Disentangling the Spatial Structure and Style in Conditional VAE

TL;DR

This paper introduces a method to disentangle spatial structure and style in conditional VAE, improving interpretability and control over generated images by separating label-relevant and irrelevant latent factors.

Contribution

It proposes a novel disentanglement approach in cVAE that separates spatial and style information, with a flexible generator architecture utilizing adaptive normalization.

Findings

Effective disentanglement demonstrated on two datasets

Improved control over spatial and style features in generated images

Enhanced interpretability of latent space representations

Abstract

This paper aims to disentangle the latent space in cVAE into the spatial structure and the style code, which are complementary to each other, with one of them $z_s$ being label relevant and the other $z_u$ irrelevant. The generator is built by a connected encoder-decoder and a label condition mapping network. Depending on whether the label is related with the spatial structure, the output $z_s$ from the condition mapping network is used either as a style code or a spatial structure code. The encoder provides the label irrelevant posterior from which $z_u$ is sampled. The decoder employs $z_s$ and $z_u$ in each layer by adaptive normalization like SPADE or AdaIN. Extensive experiments on two datasets with different types of labels show the effectiveness of our method.