Latent Transformations for Object View Points Synthesis

TL;DR

This paper introduces LTNN, a lightweight, fully-convolutional neural network for real-time object view synthesis, utilizing a novel conditional transformation unit and consistency loss to outperform existing methods across multiple tasks.

Contribution

The paper presents a new latent transformation neural network with a dedicated transformation unit and a consistency loss, improving view synthesis quality and efficiency.

Findings

Outperforms state-of-the-art in multi-view reconstruction, face view synthesis, and object rotation.

Reduces inference computational demand by 30%.

Demonstrates versatility across diverse 3D view synthesis tasks.

Abstract

We propose a fully-convolutional conditional generative model, the latent transformation neural network (LTNN), capable of view synthesis using a light-weight neural network suited for real-time applications. In contrast to existing conditional generative models which incorporate conditioning information via concatenation, we introduce a dedicated network component, the conditional transformation unit (CTU), designed to learn the latent space transformations corresponding to specified target views. In addition, a consistency loss term is defined to guide the network toward learning the desired latent space mappings, a task-divided decoder is constructed to refine the quality of generated views, and an adaptive discriminator is introduced to improve the adversarial training process. The generality of the proposed methodology is demonstrated on a collection of three diverse tasks: multi-view reconstruction on real hand depth images, view synthesis of real and synthetic faces, and the rotation of rigid objects. The proposed model is shown to exceed state-of-the-art results in each category while simultaneously achieving a reduction in the computational demand required for inference by 30% on average.