Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations

TL;DR

This paper introduces a mutual encoder-decoder CNN that jointly recovers structures and textures in image inpainting by utilizing features from different layers and applying feature equalization techniques, leading to improved hole filling performance.

Contribution

The paper proposes a novel mutual encoder-decoder architecture with feature equalization for joint structure and texture recovery in image inpainting, enhancing feature utilization.

Findings

Outperforms state-of-the-art methods on benchmark datasets.

Effectively recovers both structures and textures in inpainting tasks.

Demonstrates the benefit of feature equalization in CNN-based inpainting.

Abstract

Deep encoder-decoder based CNNs have advanced image inpainting methods for hole filling. While existing methods recover structures and textures step-by-step in the hole regions, they typically use two encoder-decoders for separate recovery. The CNN features of each encoder are learned to capture either missing structures or textures without considering them as a whole. The insufficient utilization of these encoder features limit the performance of recovering both structures and textures. In this paper, we propose a mutual encoder-decoder CNN for joint recovery of both. We use CNN features from the deep and shallow layers of the encoder to represent structures and textures of an input image, respectively. The deep layer features are sent to a structure branch and the shallow layer features are sent to a texture branch. In each branch, we fill holes in multiple scales of the CNN features. The filled CNN features from both branches are concatenated and then equalized. During feature equalization, we reweigh channel attentions first and propose a bilateral propagation activation function to enable spatial equalization. To this end, the filled CNN features of structure and texture mutually benefit each other to represent image content at all feature levels. We use the equalized feature to supplement decoder features for output image generation through skip connections. Experiments on the benchmark datasets show the proposed method is effective to recover structures and textures and performs favorably against state-of-the-art approaches.