CR-Fill: Generative Image Inpainting with Auxiliary Contexutal Reconstruction

TL;DR

CR-Fill introduces an attention-free generative inpainting method trained with an auxiliary contextual reconstruction task, improving plausibility and reducing artifacts without increasing inference complexity.

Contribution

The paper proposes a novel attention-free inpainting approach using a joint training scheme with a CR loss, eliminating the need for attention during inference.

Findings

Outperforms state-of-the-art methods quantitatively.

Produces more visually plausible inpainted images.

Reduces computational overhead during inference.

Abstract

Recent deep generative inpainting methods use attention layers to allow the generator to explicitly borrow feature patches from the known region to complete a missing region. Due to the lack of supervision signals for the correspondence between missing regions and known regions, it may fail to find proper reference features, which often leads to artifacts in the results. Also, it computes pair-wise similarity across the entire feature map during inference bringing a significant computational overhead. To address this issue, we propose to teach such patch-borrowing behavior to an attention-free generator by joint training of an auxiliary contextual reconstruction task, which encourages the generated output to be plausible even when reconstructed by surrounding regions. The auxiliary branch can be seen as a learnable loss function, i.e. named as contextual reconstruction (CR) loss, where query-reference feature similarity and reference-based reconstructor are jointly optimized with the inpainting generator. The auxiliary branch (i.e. CR loss) is required only during training, and only the inpainting generator is required during the inference. Experimental results demonstrate that the proposed inpainting model compares favourably against the state-of-the-art in terms of quantitative and visual performance.