Eye-for-an-eye: Appearance Transfer with Semantic Correspondence in Diffusion Models

TL;DR

This paper introduces a training-free appearance transfer method for diffusion models that explicitly uses dense semantic correspondence to better preserve structure and color in generated images.

Contribution

It proposes a novel approach that rearranges features based on semantic correspondence, improving appearance transfer quality without additional training.

Findings

Outperforms existing methods in structure preservation

Accurately transfers color from reference images

Works effectively even with unaligned images

Abstract

As pre-trained text-to-image diffusion models have become a useful tool for image synthesis, people want to specify the results in various ways. This paper tackles training-free appearance transfer, which produces an image with the structure of a target image from the appearance of a reference image. Existing methods usually do not reflect semantic correspondence, as they rely on query-key similarity within the self-attention layer to establish correspondences between images. To this end, we propose explicitly rearranging the features according to the dense semantic correspondences. Extensive experiments show the superiority of our method in various aspects: preserving the structure of the target and reflecting the correct color from the reference, even when the two images are not aligned.