Training-and-Prompt-Free General Painterly Harmonization via Zero-Shot Disentenglement on Style and Content References

TL;DR

The paper introduces TF-GPH, a training-and-prompt-free method for painterly image harmonization that uses novel disentanglement and reweighting mechanisms to improve style-content blending without additional training or prompts.

Contribution

It presents a new zero-shot harmonization approach with innovative disentanglement and reweighting techniques, along with a new benchmark and evaluation metrics.

Findings

Effective in harmonizing images without training or prompts

Outperforms existing methods on multiple benchmarks

Proposed metrics better reflect real-world performance

Abstract

Painterly image harmonization aims at seamlessly blending disparate visual elements within a single image. However, previous approaches often struggle due to limitations in training data or reliance on additional prompts, leading to inharmonious and content-disrupted output. To surmount these hurdles, we design a Training-and-prompt-Free General Painterly Harmonization method (TF-GPH). TF-GPH incorporates a novel ``Similarity Disentangle Mask'', which disentangles the foreground content and background image by redirecting their attention to corresponding reference images, enhancing the attention mechanism for multi-image inputs. Additionally, we propose a ``Similarity Reweighting'' mechanism to balance harmonization between stylization and content preservation. This mechanism minimizes content disruption by prioritizing the content-similar features within the given background style reference. Finally, we address the deficiencies in existing benchmarks by proposing novel range-based evaluation metrics and a new benchmark to better reflect real-world applications. Extensive experiments demonstrate the efficacy of our method in all benchmarks. More detailed in https://github.com/BlueDyee/TF-GPH.