LumiSculpt: Enabling Consistent Portrait Lighting in Video Generation

TL;DR

LumiSculpt introduces a novel method for precise, consistent lighting control in video generation, enabling customizable lighting effects through a learnable module and a new dataset, LumiHuman.

Contribution

The paper presents LumiSculpt, a new approach for independent lighting control in video generation, including a plug-and-play module and a dedicated dataset for training.

Findings

LumiSculpt achieves high-quality, precise lighting control in videos.

The LumiHuman dataset effectively supports training for lighting manipulation.

Experimental results validate the flexibility and effectiveness of LumiSculpt.

Abstract

Lighting plays a pivotal role in ensuring the naturalness and aesthetic quality of video generation. However, the impact of lighting is deeply coupled with other factors of videos, e.g., objects and scenes. Thus, it remains challenging to disentangle and model coherent lighting conditions independently, limiting the flexibility to control lighting in video generation. In this paper, inspired by the established controllable T2I models, we propose LumiSculpt, which enables precise and consistent lighting control in T2V generation models. LumiSculpt equips the video generation with new interactive capabilities, allowing the input of reference image sequences with customized lighting conditions. Furthermore, the core learnable plug-and-play module of LumiSculpt facilitates direct control over the intensity, position and trajectory of an assumed light source in video diffusion models. To effectively train LumiSculpt and address the issue of insufficient lighting data, we construct LumiHuman, a new lightweight and flexible dataset for portrait lighting of images and videos. Experimental results demonstrate that LumiSculpt achieves precise and high-quality lighting control in video generation. The analysis demonstrates the flexibility of LumiHuman.