One-Stage Inpainting with Bilateral Attention and Pyramid Filling Block

TL;DR

This paper introduces a novel single-network deep learning method for image inpainting that employs Pyramid Filling Blocks and Bilateral Attention layers to produce high-quality, structurally coherent results more efficiently than traditional two-stage models.

Contribution

The paper proposes a new single-network inpainting approach with Pyramid Filling Blocks and Bilateral Attention, reducing inference time and improving result quality over existing two-stage methods.

Findings

Outperforms existing methods on multiple datasets

Produces sharper and more structurally coherent inpainted images

Reduces inference time compared to two-stage architectures

Abstract

Recent deep learning based image inpainting methods which utilize contextual information and two-stage architecture have exhibited remarkable performance. However, the two-stage architecture is time-consuming, the contextual information lack high-level semantics and ignores both the semantic relevance and distance information of hole's feature patches, these limitations result in blurry textures and distorted structures of final result. Motivated by these observations, we propose a new deep generative model-based approach, which trains a shared network twice with different targets and utilizes a single network during the testing phase, so that we can effectively save inference time. Specifically, the targets of two training steps are structure reconstruction and texture generation respectively. During the second training, we first propose a Pyramid Filling Block (PF-block) to utilize the high-level features that the hole regions has been filled to guide the filling process of low-level features progressively, the missing content can be filled from deep to shallow in a pyramid fashion. Then, inspired by the classical bilateral filter [30], we propose the Bilateral Attention layer (BA-layer) to optimize filled feature map, which synthesizes feature patches at each position by computing weighted sums of the surrounding feature patches, these weights are derived by considering both distance and value relationships between feature patches, thus making the visually plausible inpainting results. Finally, experiments on multiple publicly available datasets show the superior performance of our approach.