Structure-Guided Image Completion with Image-level and Object-level Semantic Discriminators

TL;DR

This paper introduces a structure-guided image completion method using semantic and object-level discriminators, significantly enhancing realism and supporting diverse editing tasks, achieving state-of-the-art results.

Contribution

The work proposes a novel learning framework with semantic and object-level discriminators to improve complex object generation in image completion.

Findings

Achieves state-of-the-art results on segmentation-guided completion.

Improves realism of generated objects and scenes.

Supports multiple editing use cases including insertion and removal.

Abstract

Structure-guided image completion aims to inpaint a local region of an image according to an input guidance map from users. While such a task enables many practical applications for interactive editing, existing methods often struggle to hallucinate realistic object instances in complex natural scenes. Such a limitation is partially due to the lack of semantic-level constraints inside the hole region as well as the lack of a mechanism to enforce realistic object generation. In this work, we propose a learning paradigm that consists of semantic discriminators and object-level discriminators for improving the generation of complex semantics and objects. Specifically, the semantic discriminators leverage pretrained visual features to improve the realism of the generated visual concepts. Moreover, the object-level discriminators take aligned instances as inputs to enforce the realism of individual objects. Our proposed scheme significantly improves the generation quality and achieves state-of-the-art results on various tasks, including segmentation-guided completion, edge-guided manipulation and panoptically-guided manipulation on Places2 datasets. Furthermore, our trained model is flexible and can support multiple editing use cases, such as object insertion, replacement, removal and standard inpainting. In particular, our trained model combined with a novel automatic image completion pipeline achieves state-of-the-art results on the standard inpainting task.