Image-to-Image Translation with Diffusion Transformers and CLIP-Based Image Conditioning

TL;DR

This paper introduces a diffusion transformer framework conditioned on CLIP image embeddings for high-quality, semantically consistent image-to-image translation, offering an effective alternative to GAN-based methods.

Contribution

It proposes a novel diffusion transformer model guided by CLIP embeddings for improved image translation without text or class labels.

Findings

Achieves high-quality, semantically faithful translations

Outperforms GAN-based models on benchmark datasets

Effectively preserves structural and visual fidelity

Abstract

Image-to-image translation aims to learn a mapping between a source and a target domain, enabling tasks such as style transfer, appearance transformation, and domain adaptation. In this work, we explore a diffusion-based framework for image-to-image translation by adapting Diffusion Transformers (DiT), which combine the denoising capabilities of diffusion models with the global modeling power of transformers. To guide the translation process, we condition the model on image embeddings extracted from a pre-trained CLIP encoder, allowing for fine-grained and structurally consistent translations without relying on text or class labels. We incorporate both a CLIP similarity loss to enforce semantic consistency and an LPIPS perceptual loss to enhance visual fidelity during training. We validate our approach on two benchmark datasets: face2comics, which translates real human faces to comic-style illustrations, and edges2shoes, which translates edge maps to realistic shoe images. Experimental results demonstrate that DiT, combined with CLIP-based conditioning and perceptual similarity objectives, achieves high-quality, semantically faithful translations, offering a promising alternative to GAN-based models for paired image-to-image translation tasks.