Controllable Semantic Image Inpainting
Jin Xu, Yee Whye Teh

TL;DR
This paper introduces a deep generative model for user-controllable semantic image inpainting, allowing flexible, semantically coherent filling of missing image regions based on observed pixels.
Contribution
It presents a novel combination of an encoder, disentangled latent variables, and a bidirectional PixelCNN for controllable and coherent image inpainting.
Findings
Generates semantically plausible inpaintings matching user input
Maintains local consistency with observed pixels
Demonstrates effectiveness through extensive experiments
Abstract
We develop a method for user-controllable semantic image inpainting: Given an arbitrary set of observed pixels, the unobserved pixels can be imputed in a user-controllable range of possibilities, each of which is semantically coherent and locally consistent with the observed pixels. We achieve this using a deep generative model bringing together: an encoder which can encode an arbitrary set of observed pixels, latent variables which are trained to represent disentangled factors of variations, and a bidirectional PixelCNN model. We experimentally demonstrate that our method can generate plausible inpainting results matching the user-specified semantics, but is still coherent with observed pixels. We justify our choices of architecture and training regime through more experiments.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Advanced Image Processing Techniques · Video Analysis and Summarization
