GR3EN: Generative Relighting for 3D Environments

TL;DR

This paper introduces GR3EN, a novel method for controllable relighting of large 3D room-scale environments by leveraging diffusion models to produce high-quality, view-consistent relighting without solving complex inverse rendering problems.

Contribution

The paper presents a new approach that distills diffusion model outputs into 3D reconstructions, enabling high-quality, controllable relighting of complex real-world scenes.

Findings

Effective relighting of large 3D scenes demonstrated on synthetic data.

High fidelity and view consistency in relighted scenes validated on real-world datasets.

Avoids complex inverse rendering, simplifying 3D relighting process.

Abstract

We present a method for relighting 3D reconstructions of large room-scale environments. Existing solutions for 3D scene relighting often require solving under-determined or ill-conditioned inverse rendering problems, and are as such unable to produce high-quality results on complex real-world scenes. Though recent progress in using generative image and video diffusion models for relighting has been promising, these techniques are either limited to 2D image and video relighting or 3D relighting of individual objects. Our approach enables controllable 3D relighting of room-scale scenes by distilling the outputs of a video-to-video relighting diffusion model into a 3D reconstruction. This side-steps the need to solve a difficult inverse rendering problem, and results in a flexible system that can relight 3D reconstructions of complex real-world scenes. We validate our approach on both synthetic and real-world datasets to show that it can faithfully render novel views of scenes under new lighting conditions.