Single-Reference Text-to-Image Manipulation with Dual Contrastive Denoising Score

TL;DR

This paper introduces Dual Contrastive Denoising Score, a novel framework that enhances real image editing with text-to-image diffusion models by improving content preservation and flexibility without additional training.

Contribution

It proposes a dual contrastive loss leveraging intermediate self-attention representations, enabling effective real image editing and zero-shot translation using pretrained diffusion models.

Findings

Outperforms existing image editing methods in quality and fidelity.

Maintains structure while allowing flexible content modifications.

Operates without additional training or auxiliary networks.

Abstract

Large-scale text-to-image generative models have shown remarkable ability to synthesize diverse and high-quality images. However, it is still challenging to directly apply these models for editing real images for two reasons. First, it is difficult for users to come up with a perfect text prompt that accurately describes every visual detail in the input image. Second, while existing models can introduce desirable changes in certain regions, they often dramatically alter the input content and introduce unexpected changes in unwanted regions. To address these challenges, we present Dual Contrastive Denoising Score, a simple yet powerful framework that leverages the rich generative prior of text-to-image diffusion models. Inspired by contrastive learning approaches for unpaired image-to-image translation, we introduce a straightforward dual contrastive loss within the proposed framework. Our approach utilizes the extensive spatial information from the intermediate representations of the self-attention layers in latent diffusion models without depending on auxiliary networks. Our method achieves both flexible content modification and structure preservation between input and output images, as well as zero-shot image-to-image translation. Through extensive experiments, we show that our approach outperforms existing methods in real image editing while maintaining the capability to directly utilize pretrained text-to-image diffusion models without further training.