Deep Image Synthesis from Intuitive User Input: A Review and Perspectives

TL;DR

This paper reviews recent advances in deep generative models enabling photo-realistic image synthesis from intuitive user inputs like sketches or text, highlighting new perspectives and evaluation methods.

Contribution

It provides a comprehensive review of recent deep learning-based image synthesis methods from intuitive inputs, emphasizing input versatility, methodology, and evaluation.

Findings

Advances in GANs, VAEs, and flow-based models improve image synthesis from user inputs.

New perspectives on input representation and interactivity are emerging.

Benchmark datasets and evaluation metrics are evolving for better comparison.

Abstract

In many applications of computer graphics, art and design, it is desirable for a user to provide intuitive non-image input, such as text, sketch, stroke, graph or layout, and have a computer system automatically generate photo-realistic images that adhere to the input content. While classic works that allow such automatic image content generation have followed a framework of image retrieval and composition, recent advances in deep generative models such as generative adversarial networks (GANs), variational autoencoders (VAEs), and flow-based methods have enabled more powerful and versatile image generation tasks. This paper reviews recent works for image synthesis given intuitive user input, covering advances in input versatility, image generation methodology, benchmark datasets, and evaluation metrics. This motivates new perspectives on input representation and interactivity, cross pollination between major image generation paradigms, and evaluation and comparison of generation methods.