Controllable Semantic Image Inpainting

TL;DR

This paper introduces a deep generative model for user-controllable semantic image inpainting, allowing flexible, semantically coherent filling of missing image regions based on observed pixels.

Contribution

It presents a novel combination of an encoder, disentangled latent variables, and a bidirectional PixelCNN for controllable and coherent image inpainting.

Findings

Generates semantically plausible inpaintings matching user input

Maintains local consistency with observed pixels

Demonstrates effectiveness through extensive experiments

Abstract

We develop a method for user-controllable semantic image inpainting: Given an arbitrary set of observed pixels, the unobserved pixels can be imputed in a user-controllable range of possibilities, each of which is semantically coherent and locally consistent with the observed pixels. We achieve this using a deep generative model bringing together: an encoder which can encode an arbitrary set of observed pixels, latent variables which are trained to represent disentangled factors of variations, and a bidirectional PixelCNN model. We experimentally demonstrate that our method can generate plausible inpainting results matching the user-specified semantics, but is still coherent with observed pixels. We justify our choices of architecture and training regime through more experiments.