Region-to-Region: Enhancing Generative Image Harmonization with Adaptive Regional Injection

TL;DR

This paper introduces R2R, a novel image harmonization model that uses adaptive regional injection and a new synthetic dataset to improve detail preservation and realism in composite images.

Contribution

The paper proposes the R2R model with Clear-VAE and MACA for enhanced harmonization, and introduces the RPHarmony dataset generated via Random Poisson Blending.

Findings

R2R outperforms existing methods in quantitative metrics.

The RPHarmony dataset improves model generalization to real images.

The approach effectively preserves details and enhances visual harmony.

Abstract

The goal of image harmonization is to adjust the foreground in a composite image to achieve visual consistency with the background. Recently, latent diffusion model (LDM) are applied for harmonization, achieving remarkable results. However, LDM-based harmonization faces challenges in detail preservation and limited harmonization ability. Additionally, current synthetic datasets rely on color transfer, which lacks local variations and fails to capture complex real-world lighting conditions. To enhance harmonization capabilities, we propose the Region-to-Region transformation. By injecting information from appropriate regions into the foreground, this approach preserves original details while achieving image harmonization or, conversely, generating new composite data. From this perspective, We propose a novel model R2R. Specifically, we design Clear-VAE to preserve high-frequency details in the foreground using Adaptive Filter while eliminating disharmonious elements. To further enhance harmonization, we introduce the Harmony Controller with Mask-aware Adaptive Channel Attention (MACA), which dynamically adjusts the foreground based on the channel importance of both foreground and background regions. To address the limitation of existing datasets, we propose Random Poisson Blending, which transfers color and lighting information from a suitable region to the foreground, thereby generating more diverse and challenging synthetic images. Using this method, we construct a new synthetic dataset, RPHarmony. Experiments demonstrate the superiority of our method over other methods in both quantitative metrics and visual harmony. Moreover, our dataset helps the model generate more realistic images in real examples. Our code, dataset, and model weights have all been released for open access.