PHNet: Patch-based Normalization for Portrait Harmonization

TL;DR

This paper introduces PHNet, a patch-based normalization network for portrait harmonization that improves visual coherence in composite images, demonstrating state-of-the-art results and strong generalization across datasets.

Contribution

The paper proposes a novel patch-based normalization approach and a statistical color transfer feature extractor for improved portrait harmonization.

Findings

Achieves state-of-the-art results on iHarmony4 dataset.

Demonstrates strong generalization on a new FFHQ-based dataset.

Outperforms existing methods in visual coherence and harmonization quality.

Abstract

A common problem for composite images is the incompatibility of their foreground and background components. Image harmonization aims to solve this problem, making the whole image look more authentic and coherent. Most existing solutions predict lookup tables (LUTs) or reconstruct images, utilizing various attributes of composite images. Recent approaches have primarily focused on employing global transformations like normalization and color curve rendering to achieve visual consistency, and they often overlook the importance of local visual coherence. We present a patch-based harmonization network consisting of novel Patch-based normalization (PN) blocks and a feature extractor based on statistical color transfer. Extensive experiments demonstrate the network's high generalization capability for different domains. Our network achieves state-of-the-art results on the iHarmony4 dataset. Also, we created a new human portrait harmonization dataset based on FFHQ and checked the proposed method to show the generalization ability by achieving the best metrics on it. The benchmark experiments confirm that the suggested patch-based normalization block and feature extractor effectively improve the network's capability to harmonize portraits. Our code and model baselines are publicly available.