ImPoster: Text and Frequency Guidance for Subject Driven Action Personalization using Diffusion Models

TL;DR

ImPoster is an unsupervised diffusion-based method that personalizes images by transferring subject and action from a source to a target using frequency-guided diffusion steps, without requiring annotations.

Contribution

It introduces a novel frequency guidance technique for diffusion models enabling subject and action-driven image personalization without annotations.

Findings

Outperforms baseline methods in diverse source-driving pairs.

First approach to combine subject and action personalization in diffusion models.

Effective frequency guidance improves target image fidelity.

Abstract

We present ImPoster, a novel algorithm for generating a target image of a 'source' subject performing a 'driving' action. The inputs to our algorithm are a single pair of a source image with the subject that we wish to edit and a driving image with a subject of an arbitrary class performing the driving action, along with the text descriptions of the two images. Our approach is completely unsupervised and does not require any access to additional annotations like keypoints or pose. Our approach builds on a pretrained text-to-image latent diffusion model and learns the characteristics of the source and the driving image by finetuning the diffusion model for a small number of iterations. At inference time, ImPoster performs step-wise text prompting i.e. it denoises by first moving in the direction of the image manifold corresponding to the driving image followed by the direction of the image manifold corresponding to the text description of the desired target image. We propose a novel diffusion guidance formulation, image frequency guidance, to steer the generation towards the manifold of the source subject and the driving action at every step of the inference denoising. Our frequency guidance formulations are derived from the frequency domain properties of images. We extensively evaluate ImPoster on a diverse set of source-driving image pairs to demonstrate improvements over baselines. To the best of our knowledge, ImPoster is the first approach towards achieving both subject-driven as well as action-driven image personalization. Code and data is available at https://github.com/divyakraman/ImPosterDiffusion2024.