SCS-Co: Self-Consistent Style Contrastive Learning for Image Harmonization

TL;DR

This paper introduces SCS-Co, a novel self-consistent style contrastive learning approach for image harmonization that improves visual consistency by learning from multiple negative samples and jointly constraining style features.

Contribution

The paper proposes a new contrastive learning scheme with dynamic negative sample generation and a background-attentional normalization to enhance image harmonization quality.

Findings

Outperforms state-of-the-art methods in quantitative metrics

Produces more photorealistic harmonized images

Effectively learns style consistency from multiple negative samples

Abstract

Image harmonization aims to achieve visual consistency in composite images by adapting a foreground to make it compatible with a background. However, existing methods always only use the real image as the positive sample to guide the training, and at most introduce the corresponding composite image as a single negative sample for an auxiliary constraint, which leads to limited distortion knowledge, and further causes a too large solution space, making the generated harmonized image distorted. Besides, none of them jointly constrain from the foreground self-style and foreground-background style consistency, which exacerbates this problem. Moreover, recent region-aware adaptive instance normalization achieves great success but only considers the global background feature distribution, making the aligned foreground feature distribution biased. To address these issues, we propose a self-consistent style contrastive learning scheme (SCS-Co). By dynamically generating multiple negative samples, our SCS-Co can learn more distortion knowledge and well regularize the generated harmonized image in the style representation space from two aspects of the foreground self-style and foreground-background style consistency, leading to a more photorealistic visual result. In addition, we propose a background-attentional adaptive instance normalization (BAIN) to achieve an attention-weighted background feature distribution according to the foreground-background feature similarity. Experiments demonstrate the superiority of our method over other state-of-the-art methods in both quantitative comparison and visual analysis.