Novel View Synthesis on Unpaired Data by Conditional Deformable Variational Auto-Encoder
Mingyu Yin, Li Sun, Qingli Li

TL;DR
This paper introduces a novel view synthesis method using a conditional variational auto-encoder GAN framework that does not require paired data, employing a deformable module and adversarial training to improve view translation and disentanglement.
Contribution
It proposes a new view translation model with a conditional deformable module and deformed feature normalization, enabling unpaired view synthesis within a cVAE-GAN framework.
Findings
Effective view translation on MultiPIE and 3D chair datasets
The proposed modules improve disentanglement of view and other factors
Ablation studies confirm the effectiveness of the framework
Abstract
Novel view synthesis often needs the paired data from both the source and target views. This paper proposes a view translation model under cVAE-GAN framework without requiring the paired data. We design a conditional deformable module (CDM) which uses the view condition vectors as the filters to convolve the feature maps of the main branch in VAE. It generates several pairs of displacement maps to deform the features, like the 2D optical flows. The results are fed into the deformed feature based normalization module (DFNM), which scales and offsets the main branch feature, given its deformed one as the input from the side branch. Taking the advantage of the CDM and DFNM, the encoder outputs a view-irrelevant posterior, while the decoder takes the code drawn from it to synthesize the reconstructed and the viewtranslated images. To further ensure the disentanglement between the views and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Vision and Imaging · Generative Adversarial Networks and Image Synthesis · Computer Graphics and Visualization Techniques
MethodsConditional Variational Auto Encoder · USD Coin Customer Service Number +1-833-534-1729
