Deep Portrait Image Completion and Extrapolation

TL;DR

This paper introduces a two-stage deep learning framework for accurate portrait image completion and extrapolation, combining structure recovery and image synthesis to improve results over existing methods.

Contribution

The proposed method uniquely integrates human structure parsing with image completion, enabling realistic portrait editing and extrapolation, and demonstrates applicability to other image types.

Findings

Outperforms state-of-the-art image completion methods on portrait datasets.

Enables applications like occlusion removal and portrait extrapolation.

Applicable to animal images beyond human portraits.

Abstract

General image completion and extrapolation methods often fail on portrait images where parts of the human body need to be recovered - a task that requires accurate human body structure and appearance synthesis. We present a two-stage deep learning framework for tacking this problem. In the first stage, given a portrait image with an incomplete human body, we extract a complete, coherent human body structure through a human parsing network, which focuses on structure recovery inside the unknown region with the help of pose estimation. In the second stage, we use an image completion network to fill the unknown region, guided by the structure map recovered in the first stage. For realistic synthesis the completion network is trained with both perceptual loss and conditional adversarial loss. We evaluate our method on public portrait image datasets, and show that it outperforms other state-of-art general image completion methods. Our method enables new portrait image editing applications such as occlusion removal and portrait extrapolation. We further show that the proposed general learning framework can be applied to other types of images, e.g. animal images.