IterGANs: Iterative GANs to Learn and Control 3D Object Transformation

TL;DR

IterGANs introduce an iterative GAN framework that learns to manipulate and transform 3D object representations from a single 2D image, enabling controllable and generalizable object transformations.

Contribution

This paper presents a novel iterative GAN approach that learns implicit 3D and appearance models from a single image, allowing for controllable object transformations and generalization to unseen objects.

Findings

IterGANs successfully generate rotated object images from a single input.

The iterative process improves transformation accuracy and quality.

Intermediate generated images provide additional supervision signals.

Abstract

We are interested in learning visual representations which allow for 3D manipulations of visual objects based on a single 2D image. We cast this into an image-to-image transformation task, and propose Iterative Generative Adversarial Networks (IterGANs) which iteratively transform an input image into an output image. Our models learn a visual representation that can be used for objects seen in training, but also for never seen objects. Since object manipulation requires a full understanding of the geometry and appearance of the object, our IterGANs learn an implicit 3D model and a full appearance model of the object, which are both inferred from a single (test) image. Two advantages of IterGANs are that the intermediate generated images can be used for an additional supervision signal, even in an unsupervised fashion, and that the number of iterations can be used as a control signal to steer the transformation. Experiments on rotated objects and scenes show how IterGANs help with the generation process.