Region Normalization for Image Inpainting

TL;DR

This paper introduces Region Normalization (RN), a novel feature normalization technique that divides image regions based on corruption masks to improve neural network training for image inpainting, outperforming existing methods.

Contribution

The paper proposes a new region-wise normalization method, RN, that addresses mean and variance shifts caused by full-spatial normalization in image inpainting networks, with two variants RN-B and RN-L.

Findings

RN outperforms state-of-the-art methods in image inpainting.

RN improves training stability and results across different networks.

The method generalizes well to various inpainting architectures.

Abstract

Feature Normalization (FN) is an important technique to help neural network training, which typically normalizes features across spatial dimensions. Most previous image inpainting methods apply FN in their networks without considering the impact of the corrupted regions of the input image on normalization, e.g. mean and variance shifts. In this work, we show that the mean and variance shifts caused by full-spatial FN limit the image inpainting network training and we propose a spatial region-wise normalization named Region Normalization (RN) to overcome the limitation. RN divides spatial pixels into different regions according to the input mask, and computes the mean and variance in each region for normalization. We develop two kinds of RN for our image inpainting network: (1) Basic RN (RN-B), which normalizes pixels from the corrupted and uncorrupted regions separately based on the original inpainting mask to solve the mean and variance shift problem; (2) Learnable RN (RN-L), which automatically detects potentially corrupted and uncorrupted regions for separate normalization, and performs global affine transformation to enhance their fusion. We apply RN-B in the early layers and RN-L in the latter layers of the network respectively. Experiments show that our method outperforms current state-of-the-art methods quantitatively and qualitatively. We further generalize RN to other inpainting networks and achieve consistent performance improvements. Our code is available at https://github.com/geekyutao/RN.