GenLit: Reformulating Single-Image Relighting as Video Generation

TL;DR

GenLit leverages a video diffusion model to relight single images by manipulating virtual light sources, bypassing traditional 3D reconstruction and rendering, and producing realistic relighting effects in video sequences.

Contribution

This work introduces GenLit, a novel framework that uses a video diffusion model to perform single-image relighting by directly manipulating virtual light sources, avoiding explicit 3D asset reconstruction.

Findings

Model fine-tuned on synthetic data generalizes to real scenes.

Produces plausible shadows and inter-reflections.

Enables relighting without explicit 3D reconstruction or ray-tracing.

Abstract

Manipulating the illumination of a 3D scene within a single image represents a fundamental challenge in computer vision and graphics. This problem has traditionally been addressed using inverse rendering techniques, which involve explicit 3D asset reconstruction and costly ray-tracing simulations. Meanwhile, recent advancements in visual foundation models suggest that a new paradigm could soon be possible -- one that replaces explicit physical models with networks that are trained on large amounts of image and video data. In this paper, we exploit the implicit scene understanding of a video diffusion model, particularly Stable Video Diffusion, to relight a single image. We introduce GenLit, a framework that distills the ability of a graphics engine to perform light manipulation into a video-generation model, enabling users to directly insert and manipulate a point light in the 3D world within a given image and generate results directly as a video sequence. We find that a model fine-tuned on only a small synthetic dataset generalizes to real-world scenes, enabling single-image relighting with plausible and convincing shadows and inter-reflections. Our results highlight the ability of video foundation models to capture rich information about lighting, material, and shape, and our findings indicate that such models, with minimal training, can be used to perform relighting without explicit asset reconstruction or ray-tracing. . Project page: https://genlit.is.tue.mpg.de/.