Seed-to-Seed: Image Translation in Diffusion Seed Space

TL;DR

This paper presents Seed-to-Seed (StS), a novel diffusion model-based method for unpaired image-to-image translation that leverages the semantic information in seed-space for structure-preserving transformations.

Contribution

It introduces a new seed-space manipulation technique for image translation using a trained sts-GAN and diffusion models, enabling structure-preserving unpaired image translation.

Findings

Outperforms existing GAN and diffusion-based methods in automotive scene translation

Uses seed-space for effective semantic and structural image manipulation

Demonstrates versatility across various unpaired image translation tasks

Abstract

We introduce Seed-to-Seed Translation (StS), a novel approach for Image-to-Image Translation using diffusion models (DMs), aimed at translations that require close adherence to the structure of the source image. In contrast to existing methods that modify images during the diffusion sampling process, we leverage the semantic information encoded within the space of inverted seeds of a pretrained DM, dubbed as the seed-space. We demonstrate that inverted seeds can be used for discriminative tasks, and can also be manipulated to achieve desired transformations in an unpaired image-to-image translation setting. Our method involves training an sts-GAN, an unpaired translation model between source and target seeds, based on CycleGAN. The final translated images are obtained by initiating the DM's sampling process from the translated seeds. A ControlNet is used to ensure the structural preservation of the input image. We demonstrate the effectiveness of our approach for the task of translating automotive scenes, showcasing superior performance compared to existing GAN-based and diffusion-based methods, as well as for several other unpaired image translation tasks. Our approach offers a fresh perspective on leveraging the semantic information encoded within the seed-space of pretrained DMs for effective image editing and manipulation.