Novel View Synthesis on Unpaired Data by Conditional Deformable Variational Auto-Encoder

TL;DR

This paper introduces a novel view synthesis method using a conditional variational auto-encoder GAN framework that does not require paired data, employing a deformable module and adversarial training to improve view translation and disentanglement.

Contribution

It proposes a new view translation model with a conditional deformable module and deformed feature normalization, enabling unpaired view synthesis within a cVAE-GAN framework.

Findings

Effective view translation on MultiPIE and 3D chair datasets

The proposed modules improve disentanglement of view and other factors

Ablation studies confirm the effectiveness of the framework

Abstract

Novel view synthesis often needs the paired data from both the source and target views. This paper proposes a view translation model under cVAE-GAN framework without requiring the paired data. We design a conditional deformable module (CDM) which uses the view condition vectors as the filters to convolve the feature maps of the main branch in VAE. It generates several pairs of displacement maps to deform the features, like the 2D optical flows. The results are fed into the deformed feature based normalization module (DFNM), which scales and offsets the main branch feature, given its deformed one as the input from the side branch. Taking the advantage of the CDM and DFNM, the encoder outputs a view-irrelevant posterior, while the decoder takes the code drawn from it to synthesize the reconstructed and the viewtranslated images. To further ensure the disentanglement between the views and other factors, we add adversarial training on the code. The results and ablation studies on MultiPIE and 3D chair datasets validate the effectiveness of the framework in cVAE and the designed module.