Perceptual Artifacts Localization for Inpainting

TL;DR

This paper introduces a method for automatically localizing perceptual artifacts in inpainted images, enabling better evaluation and iterative refinement of inpainting models, with demonstrated improvements in artifact reduction and image quality.

Contribution

It presents a new artifact segmentation dataset, a segmentation network for artifact localization, and a perceptual artifact ratio metric for evaluating inpainting quality.

Findings

Reliable artifact localization in inpainted images

PAR correlates strongly with user preferences

Iterative refinement reduces artifacts and improves inpainting quality

Abstract

Image inpainting is an essential task for multiple practical applications like object removal and image editing. Deep GAN-based models greatly improve the inpainting performance in structures and textures within the hole, but might also generate unexpected artifacts like broken structures or color blobs. Users perceive these artifacts to judge the effectiveness of inpainting models, and retouch these imperfect areas to inpaint again in a typical retouching workflow. Inspired by this workflow, we propose a new learning task of automatic segmentation of inpainting perceptual artifacts, and apply the model for inpainting model evaluation and iterative refinement. Specifically, we first construct a new inpainting artifacts dataset by manually annotating perceptual artifacts in the results of state-of-the-art inpainting models. Then we train advanced segmentation networks on this dataset to reliably localize inpainting artifacts within inpainted images. Second, we propose a new interpretable evaluation metric called Perceptual Artifact Ratio (PAR), which is the ratio of objectionable inpainted regions to the entire inpainted area. PAR demonstrates a strong correlation with real user preference. Finally, we further apply the generated masks for iterative image inpainting by combining our approach with multiple recent inpainting methods. Extensive experiments demonstrate the consistent decrease of artifact regions and inpainting quality improvement across the different methods.