Latent Gaussian Splatting for 4D Panoptic Occupancy Tracking

TL;DR

This paper introduces LaGS, a novel method for 4D panoptic occupancy tracking that combines camera-based tracking with multi-view occupancy prediction using a latent Gaussian splatting approach, achieving state-of-the-art results.

Contribution

The paper proposes a new latent Gaussian splatting technique for efficient multi-view 3D scene aggregation in 4D occupancy tracking, integrating end-to-end tracking and segmentation.

Findings

Achieves state-of-the-art performance on Occ3D nuScenes and Waymo datasets.

Introduces a sparse, point-centric latent representation for 3D scenes.

Demonstrates effective multi-view information aggregation into 3D voxel grids.

Abstract

Capturing 4D spatiotemporal surroundings is crucial for the safe and reliable operation of robots in dynamic environments. However, most existing methods address only one side of the problem: they either provide coarse geometric tracking via bounding boxes, or detailed 3D structures like voxel-based occupancy that lack explicit temporal association. In this work, we present Latent Gaussian Splatting for 4D Panoptic Occupancy Tracking (LaGS) that advances spatiotemporal scene understanding in a holistic direction. Our approach incorporates camera-based end-to-end tracking with mask-based multi-view panoptic occupancy prediction, and addresses the key challenge of efficiently aggregating multi-view information into 3D voxel grids via a novel latent Gaussian splatting approach. Specifically, we first fuse observations into 3D Gaussians that serve as a sparse point-centric latent representation of the 3D scene, and then splat the aggregated features onto a 3D voxel grid that is decoded by a mask-based segmentation head. We evaluate LaGS on the Occ3D nuScenes and Waymo datasets, achieving state-of-the-art performance for 4D panoptic occupancy tracking. We make our code available at https://lags.cs.uni-freiburg.de/.