Retrieval Augmented Image Harmonization

TL;DR

The paper introduces Raiha, a retrieval-augmented framework for image harmonization that improves results by finding relevant reference images and better understanding image content, addressing issues of ill-posedness and irrelevant attention.

Contribution

It proposes a novel retrieval-augmented approach with an efficient retrieval method and content priors, enhancing image harmonization performance significantly.

Findings

Improved harmonization quality with retrieval-augmented method

Effective retrieval of reference images with similar objects and illumination

Enhanced performance over existing methods in both non-reference and retrieval-augmented settings

Abstract

When embedding objects (foreground) into images (background), considering the influence of photography conditions like illumination, it is usually necessary to perform image harmonization to make the foreground object coordinate with the background image in terms of brightness, color, and etc. Although existing image harmonization methods have made continuous efforts toward visually pleasing results, they are still plagued by two main issues. Firstly, the image harmonization becomes highly ill-posed when there are no contents similar to the foreground object in the background, making the harmonization results unreliable. Secondly, even when similar contents are available, the harmonization process is often interfered with by irrelevant areas, mainly attributed to an insufficient understanding of image contents and inaccurate attention. As a remedy, we present a retrieval-augmented image harmonization (Raiha) framework, which seeks proper reference images to reduce the ill-posedness and restricts the attention to better utilize the useful information. Specifically, an efficient retrieval method is designed to find reference images that contain similar objects as the foreground while the illumination is consistent with the background. For training the Raiha framework to effectively utilize the reference information, a data augmentation strategy is delicately designed by leveraging existing non-reference image harmonization datasets. Besides, the image content priors are introduced to ensure reasonable attention. With the presented Raiha framework, the image harmonization performance is greatly boosted under both non-reference and retrieval-augmented settings. The source code and pre-trained models will be publicly available.