Detail Enhanced Gaussian Splatting for Large-Scale Volumetric Capture

TL;DR

This paper introduces a novel volumetric capture and rendering pipeline that combines Dynamic Gaussian Splatting with diffusion-based detail enhancement to produce high-resolution, realistic 4K facial closeups in large-scale multi-performer performances.

Contribution

It presents a new system integrating 4D Gaussian Splatting and diffusion-based detail enhancement for high-quality large-scale volumetric capture and rendering.

Findings

Achieves 4K facial detail in volumetric videos.

Improves reconstruction quality and dynamic range.

Bridges the gap between large-scale capture and high-resolution media standards.

Abstract

We present a unique system for large-scale, multi-performer, high resolution 4D volumetric capture providing realistic free-viewpoint video up to and including 4K resolution facial closeups. To achieve this, we employ a novel volumetric capture, reconstruction and rendering pipeline based on Dynamic Gaussian Splatting and Diffusion-based Detail Enhancement. We design our pipeline specifically to meet the demands of high-end media production. We employ two capture rigs: the Scene Rig, which captures multi-actor performances at a resolution which falls short of 4K production quality, and the Face Rig, which records high-fidelity single-actor facial detail to serve as a reference for detail enhancement. We first reconstruct dynamic performances from the Scene Rig using 4D Gaussian Splatting, incorporating new model designs and training strategies to improve reconstruction, dynamic range, and rendering quality. Then to render high-quality images for facial closeups, we introduce a diffusion-based detail enhancement model. This model is fine-tuned with high-fidelity data from the same actors recorded in the Face Rig. We train on paired data generated from low- and high-quality Gaussian Splatting (GS) models, using the low-quality input to match the quality of the Scene Rig, with the high-quality GS as ground truth. Our results demonstrate the effectiveness of this pipeline in bridging the gap between the scalable performance capture of a large-scale rig and the high-resolution standards required for film and media production.