Survey on Controlable Image Synthesis with Deep Learning

TL;DR

This survey reviews recent deep learning methods for 3D controllable image synthesis, focusing on geometric and photometric control, datasets, evaluation metrics, and practical applications.

Contribution

It provides a comprehensive overview of recent advances in 3D controllable image synthesis, highlighting datasets, evaluation metrics, and categorizing methods by control type.

Findings

Summarizes datasets and evaluation indicators for 3D controllable image synthesis.

Categorizes state-of-the-art methods into viewpoint/pose and structure/shape controllable synthesis.

Includes a review of photometric controllable synthesis and related applications.

Abstract

Image synthesis has attracted emerging research interests in academic and industry communities. Deep learning technologies especially the generative models greatly inspired controllable image synthesis approaches and applications, which aim to generate particular visual contents with latent prompts. In order to further investigate low-level controllable image synthesis problem which is crucial for fine image rendering and editing tasks, we present a survey of some recent works on 3D controllable image synthesis using deep learning. We first introduce the datasets and evaluation indicators for 3D controllable image synthesis. Then, we review the state-of-the-art research for geometrically controllable image synthesis in two aspects: 1) Viewpoint/pose-controllable image synthesis; 2) Structure/shape-controllable image synthesis. Furthermore, the photometrically controllable image synthesis approaches are also reviewed for 3D re-lighting researches. While the emphasis is on 3D controllable image synthesis algorithms, the related applications, products and resources are also briefly summarized for practitioners.