Splatography: Sparse multi-view dynamic Gaussian Splatting for filmmaking challenges

TL;DR

This paper introduces a novel Gaussian Splatting approach tailored for sparse multi-view video in filmmaking, effectively capturing complex dynamic scenes with limited camera data.

Contribution

It proposes splitting foreground and background representations with sparse masks, enabling high-quality dynamic 3D reconstructions without dense supervision.

Findings

Achieves up to 3 PSNR higher than state-of-the-art methods.

Produces segmented dynamic reconstructions including transparent textures.

Operates effectively with sparse camera configurations and limited supervision.

Abstract

Deformable Gaussian Splatting (GS) accomplishes photorealistic dynamic 3-D reconstruction from dense multi-view video (MVV) by learning to deform a canonical GS representation. However, in filmmaking, tight budgets can result in sparse camera configurations, which limits state-of-the-art (SotA) methods when capturing complex dynamic features. To address this issue, we introduce an approach that splits the canonical Gaussians and deformation field into foreground and background components using a sparse set of masks for frames at t=0. Each representation is separately trained on different loss functions during canonical pre-training. Then, during dynamic training, different parameters are modeled for each deformation field following common filmmaking practices. The foreground stage contains diverse dynamic features so changes in color, position and rotation are learned. While, the background containing film-crew and equipment, is typically dimmer and less dynamic so only changes in point position are learned. Experiments on 3-D and 2.5-D entertainment datasets show that our method produces SotA qualitative and quantitative results; up to 3 PSNR higher with half the model size on 3-D scenes. Unlike the SotA and without the need for dense mask supervision, our method also produces segmented dynamic reconstructions including transparent and dynamic textures. Code and video comparisons are available online: https://azzarelli.github.io/splatographypage/index.html