Deep Image Harmonization with Learnable Augmentation

TL;DR

This paper introduces SycoNet, a learnable augmentation method that enhances small-scale datasets for image harmonization by adaptively generating synthetic composites, leading to improved performance.

Contribution

The paper proposes a novel learnable augmentation technique, SycoNet, that adaptively enriches illumination diversity in small datasets for better image harmonization.

Findings

Learnable augmentation improves harmonization results.

SycoNet outperforms existing methods on benchmark datasets.

Adaptive synthetic composite generation enhances training data diversity.

Abstract

The goal of image harmonization is adjusting the foreground appearance in a composite image to make the whole image harmonious. To construct paired training images, existing datasets adopt different ways to adjust the illumination statistics of foregrounds of real images to produce synthetic composite images. However, different datasets have considerable domain gap and the performances on small-scale datasets are limited by insufficient training data. In this work, we explore learnable augmentation to enrich the illumination diversity of small-scale datasets for better harmonization performance. In particular, our designed SYthetic COmposite Network (SycoNet) takes in a real image with foreground mask and a random vector to learn suitable color transformation, which is applied to the foreground of this real image to produce a synthetic composite image. Comprehensive experiments demonstrate the effectiveness of our proposed learnable augmentation for image harmonization. The code of SycoNet is released at https://github.com/bcmi/SycoNet-Adaptive-Image-Harmonization.