Illumination and Shadows in Head Rotation: experiments with Denoising Diffusion Models

TL;DR

This paper explores manipulating illumination and head rotation in images using denoising diffusion models by analyzing latent space trajectories, enabling realistic rotations under varying lighting without retraining the model.

Contribution

It introduces a novel method to control head rotation and lighting in images by computing trajectories in the latent space of pre-trained diffusion models, without additional training.

Findings

Achieves ±30° head rotation manipulation under different lighting conditions.

Uses labels from CelebA to improve lighting variation handling.

Demonstrates the potential of latent space trajectories for image editing.

Abstract

Accurately modeling the effects of illumination and shadows during head rotation is critical in computer vision for enhancing image realism and reducing artifacts. This study delves into the latent space of denoising diffusion models to identify compelling trajectories that can express continuous head rotation under varying lighting conditions. A key contribution of our work is the generation of additional labels from the CelebA dataset,categorizing images into three groups based on prevalent illumination direction: left, center, and right. These labels play a crucial role in our approach, enabling more precise manipulations and improved handling of lighting variations. Leveraging a recent embedding technique for Denoising Diffusion Implicit Models (DDIM), our method achieves noteworthy manipulations, encompassing a wide rotation angle of $\pm 30$ degrees, while preserving individual distinct characteristics even under challenging illumination conditions. Our methodology involves computing trajectories that approximate clouds of latent representations of dataset samples with different yaw rotations through linear regression. Specific trajectories are obtained by analyzing subsets of data that share significant attributes with the source image, including light direction. Notably, our approach does not require any specific training of the generative model for the task of rotation; we merely compute and follow specific trajectories in the latent space of a pre-trained face generation model. This article showcases the potential of our approach and its current limitations through a qualitative discussion of notable examples. This study contributes to the ongoing advancements in representation learning and the semantic investigation of the latent space of generative models.