Photorealistic and Identity-Preserving Image-Based Emotion Manipulation with Latent Diffusion Models

TL;DR

This paper explores the use of latent diffusion models for realistic, identity-preserving emotion manipulation in images, demonstrating superior quality and realism over GAN-based methods through extensive evaluations.

Contribution

It introduces a novel approach combining latent diffusion models and CLIP-based text manipulation for emotion editing in in-the-wild images.

Findings

Outperforms GAN-based methods in image quality and realism

Achieves competitive emotion translation results

Provides publicly available code for reproducibility

Abstract

In this paper, we investigate the emotion manipulation capabilities of diffusion models with "in-the-wild" images, a rather unexplored application area relative to the vast and rapidly growing literature for image-to-image translation tasks. Our proposed method encapsulates several pieces of prior work, with the most important being Latent Diffusion models and text-driven manipulation with CLIP latents. We conduct extensive qualitative and quantitative evaluations on AffectNet, demonstrating the superiority of our approach in terms of image quality and realism, while achieving competitive results relative to emotion translation compared to a variety of GAN-based counterparts. Code is released as a publicly available repo.