Relightable Gaussian Splatting for Virtual Production Using Image-Based Illumination

TL;DR

This paper introduces a novel Gaussian Splatting-based framework for 3D reconstruction and relighting in virtual production, leveraging background imagery to enable flexible, high-quality scene editing without environment maps.

Contribution

It presents a VP-specific method that decomposes scenes into appearance and lighting components, capturing reflections and refractions implicitly, improving reconstruction and relighting quality.

Findings

Achieves higher-quality 3D reconstruction compared to baselines.

Enables controllable relighting using background-conditioned Gaussian Splatting.

Operates efficiently with low resource requirements and real-time rendering capabilities.

Abstract

Virtual production (VP) use LED walls to provide both background imagery and image-based lighting. While this enables on-set compositing, it couples lighting to background and scene appearance, limiting flexibility for downstream editing. In addition, inverse rendering conventionally relies on physically-based rendering to estimates 3D geometry and lighting, using environment maps. However, these maps are typically low-resolution and assume far-field lighting. In VP, with near-field and high-resolution image-based lighting, this can lead to inaccuracies and introduce complexities when editing. Addressing this, we propose a VP-specific framework for 3D reconstruction and relighting using Gaussian Splatting. This uses the known background imagery to condition the relighting process. This avoids relying on environment maps and reduces compositing to a background-image editing task. To realize our framework, we introduce a process (and associated dataset) that captures real VP scenes under varying background content and illumination conditions. This data is used to decompose a 3D scene into fixed appearance and variable lighting components. The variable lighting process simulates light transport by parameterizing each primitive with a UV coordinate, intensity value and resolution modifier. Using mipmaps, these directly sample the background texture in image space - implicitly capturing reflections and refractions without physically-based rendering. Combined with the fixed appearance component, this allows us to render relit scenes using a Gaussian Splatting rasterizer. Compared to baselines, our approach achieves higher-quality 3D reconstruction and controllable relighting. The method is efficient (<3 GB RAM, <5 GB VRAM, <2 hours training, ~35 FPS) and supports rendering useful arbitrary output variables including depth, lighting intensity, lighting color, and unlit renders.