GeoRelight: Learning Joint Geometrical Relighting and Reconstruction with Flexible Multi-Modal Diffusion Transformers

TL;DR

GeoRelight introduces a unified diffusion transformer that jointly estimates 3D geometry and relights a person from a single photo, improving physical consistency and performance.

Contribution

It proposes a novel multi-modal diffusion transformer with a new 3D representation and training method for joint geometry estimation and relighting.

Findings

Outperforms sequential and previous geometry-ignoring methods.

Uses isotropic NDC-Orthographic Depth for distortion-free 3D representation.

Employs mixed synthetic and real data for training.

Abstract

Relighting a person from a single photo is an attractive but ill-posed task, as a 2D image ambiguously entangles 3D geometry, intrinsic appearance, and illumination. Current methods either use sequential pipelines that suffer from error accumulation, or they do not explicitly leverage 3D geometry during relighting, which limits physical consistency. Since relighting and estimation of 3D geometry are mutually beneficial tasks, we propose a unified Multi-Modal Diffusion Transformer (DiT) that jointly solves for both: GeoRelight. We make this possible through two key technical contributions: isotropic NDC-Orthographic Depth (iNOD), a distortion-free 3D representation compatible with latent diffusion models; and a strategic mixed-data training method that combines synthetic and auto-labeled real data. By solving geometry and relighting jointly, GeoRelight achieves better performance than both sequential models and previous systems that ignored geometry.