Consistent Instance Field for Dynamic Scene Understanding

TL;DR

This paper presents a novel continuous probabilistic spatio-temporal representation called Consistent Instance Field for dynamic scene understanding, improving consistency and accuracy over prior discrete methods.

Contribution

It introduces a deformable 3D Gaussian-based instance embedding that disentangles visibility from identity, enabling consistent 4D scene representations from RGB images.

Findings

Outperforms state-of-the-art on HyperNeRF and Neu3D datasets.

Enables accurate novel-view panoptic segmentation.

Supports open-vocabulary 4D querying.

Abstract

We introduce Consistent Instance Field, a continuous and probabilistic spatio-temporal representation for dynamic scene understanding. Unlike prior methods that rely on discrete tracking or view-dependent features, our approach disentangles visibility from persistent object identity by modeling each space-time point with an occupancy probability and a conditional instance distribution. To realize this, we introduce a novel instance-embedded representation based on deformable 3D Gaussians, which jointly encode radiance and semantic information and are learned directly from input RGB images and instance masks through differentiable rasterization. Furthermore, we introduce new mechanisms to calibrate per-Gaussian identities and resample Gaussians toward semantically active regions, ensuring consistent instance representations across space and time. Experiments on HyperNeRF and Neu3D datasets demonstrate that our method significantly outperforms state-of-the-art methods on novel-view panoptic segmentation and open-vocabulary 4D querying tasks.