A Variational U-Net for Conditional Appearance and Shape Generation

TL;DR

This paper introduces a conditional U-Net model that combines shape and appearance modeling for image synthesis, enabling flexible and realistic image generation and transfer without needing multiple object samples.

Contribution

It proposes a novel end-to-end trainable model that separately encodes shape and appearance, improving conditional image generation and transfer capabilities.

Findings

Significant improvements over state-of-the-art methods.

Effective shape-guided image generation and transfer.

Ability to sample appearance while preserving shape.

Abstract

Deep generative models have demonstrated great performance in image synthesis. However, results deteriorate in case of spatial deformations, since they generate images of objects directly, rather than modeling the intricate interplay of their inherent shape and appearance. We present a conditional U-Net for shape-guided image generation, conditioned on the output of a variational autoencoder for appearance. The approach is trained end-to-end on images, without requiring samples of the same object with varying pose or appearance. Experiments show that the model enables conditional image generation and transfer. Therefore, either shape or appearance can be retained from a query image, while freely altering the other. Moreover, appearance can be sampled due to its stochastic latent representation, while preserving shape. In quantitative and qualitative experiments on COCO, DeepFashion, shoes, Market-1501 and handbags, the approach demonstrates significant improvements over the state-of-the-art.