Shape-guided Object Inpainting

TL;DR

This paper introduces a new shape-guided object inpainting task and proposes CogNet, a novel two-stream neural network that generates realistic objects fitting the context based on shape guidance.

Contribution

It presents a new data preparation method and a novel two-stream architecture, CogNet, specifically designed for shape-guided object inpainting, addressing limitations of previous background-focused methods.

Findings

CogNet generates realistic, contextually fitting objects.

The method outperforms existing inpainting techniques on object inpainting tasks.

The approach effectively combines bottom-up and top-down processes for object generation.

Abstract

Previous works on image inpainting mainly focus on inpainting background or partially missing objects, while the problem of inpainting an entire missing object remains unexplored. This work studies a new image inpainting task, i.e. shape-guided object inpainting. Given an incomplete input image, the goal is to fill in the hole by generating an object based on the context and implicit guidance given by the hole shape. Since previous methods for image inpainting are mainly designed for background inpainting, they are not suitable for this task. Therefore, we propose a new data preparation method and a novel Contextual Object Generator (CogNet) for the object inpainting task. On the data side, we incorporate object priors into training data by using object instances as holes. The CogNet has a two-stream architecture that combines the standard bottom-up image completion process with a top-down object generation process. A predictive class embedding module bridges the two streams by predicting the class of the missing object from the bottom-up features, from which a semantic object map is derived as the input of the top-down stream. Experiments demonstrate that the proposed method can generate realistic objects that fit the context in terms of both visual appearance and semantic meanings. Code can be found at the project page \url{https://zengxianyu.github.io/objpaint}