Director: Instance-aware Gaussian Splatting for Dynamic Scene Modeling and Understanding

TL;DR

Director introduces a unified 4D Gaussian representation for dynamic scene modeling that incorporates instance-level semantics and language alignment, improving scene decomposition and understanding.

Contribution

It presents a novel spatio-temporal Gaussian model that integrates semantics and language supervision for stable, instance-aware dynamic scene reconstruction.

Findings

Achieves temporally coherent 4D scene reconstructions.

Enables instance segmentation and open-vocabulary querying.

Reduces drift and enhances temporal stability in dynamic scenes.

Abstract

Volumetric video seeks to model dynamic scenes as temporally coherent 4D representations. While recent Gaussian-based approaches achieve impressive rendering fidelity, they primarily emphasize appearance but are largely agnostic to instance-level structure, limiting stable tracking and semantic reasoning in highly dynamic scenarios. In this paper, we present Director, a unified spatio-temporal Gaussian representation that jointly models human performance, high-fidelity rendering, and instance-level semantics. Our key insight is that embedding instance-consistent semantics naturally complements 4D modeling, enabling more accurate scene decomposition while supporting robust dynamic scene understanding. To this end, we leverage temporally aligned instance masks and sentence embeddings derived from Multimodal Large Language Models to supervise the learnable semantic features of each Gaussian via two MLP decoders, enabling language-aligned 4D representations and enforcing identity consistency over time. To enhance temporal stability, we bridge 2D optical flow with 4D Gaussians and finetune their motions, yielding reliable initialization and reducing drift. For the training, we further introduce a geometry-aware SDF constraints, along with regularization terms that enforces surface continuity, enhancing temporal coherence in dynamic foreground modeling. Experiments demonstrate that Director achieves temporally coherent 4D reconstructions while simultaneously enabling instance segmentation and open-vocabulary querying.