PanopticQuery: Unified Query-Time Reasoning for 4D Scenes

TL;DR

PanopticQuery is a unified framework that enhances 4D scene understanding by integrating high-fidelity reconstruction with semantic reasoning for complex language queries.

Contribution

It introduces a novel query-time reasoning method combining 4D Gaussian Splatting and multi-view semantic consensus for dynamic scene analysis.

Findings

Sets new state-of-the-art on complex language queries in 4D scenes.

Effectively handles attributes, actions, spatial relations, and interactions.

Provides a new benchmark, Panoptic-L4D, for language-based dynamic scene querying.

Abstract

Understanding dynamic 4D environments through natural language queries requires not only accurate scene reconstruction but also robust semantic grounding across space, time, and viewpoints. While recent methods using neural representations have advanced 4D reconstruction, they remain limited in contextual reasoning, especially for complex semantics such as interactions, temporal actions, and spatial relations. A key challenge lies in transforming noisy, view-dependent predictions into globally consistent 4D interpretations. We introduce PanopticQuery, a framework for unified query-time reasoning in 4D scenes. Our approach builds on 4D Gaussian Splatting for high-fidelity dynamic reconstruction and introduces a multi-view semantic consensus mechanism that grounds natural language queries by aggregating 2D semantic predictions across multiple views and time frames. This process filters inconsistent outputs, enforces geometric consistency, and lifts 2D semantics into structured 4D groundings via neural field optimization. To support evaluation, we present Panoptic-L4D, a new benchmark for language-based querying in dynamic scenes. Experiments demonstrate that PanopticQuery sets a new state of the art on complex language queries, effectively handling attributes, actions, spatial relationships, and multi-object interactions. A video demonstration is available in the supplementary materials.