Next Best View Selections for Semantic and Dynamic 3D Gaussian Splatting

TL;DR

This paper introduces an active learning approach using Fisher Information for selecting the most informative views to improve semantic and dynamic scene understanding in 3D Gaussian splatting, outperforming heuristic methods.

Contribution

It proposes a novel view selection algorithm based on Fisher Information that jointly handles semantic and dynamic scene modeling, advancing active learning in 3D scene understanding.

Findings

Improves rendering quality and semantic segmentation performance.

Outperforms random and heuristic view selection methods.

Effective on large-scale static and dynamic datasets.

Abstract

Understanding semantics and dynamics has been crucial for embodied agents in various tasks. Both tasks have much more data redundancy than the static scene understanding task. We formulate the view selection problem as an active learning problem, where the goal is to prioritize frames that provide the greatest information gain for model training. To this end, we propose an active learning algorithm with Fisher Information that quantifies the informativeness of candidate views with respect to both semantic Gaussian parameters and deformation networks. This formulation allows our method to jointly handle semantic reasoning and dynamic scene modeling, providing a principled alternative to heuristic or random strategies. We evaluate our method on large-scale static images and dynamic video datasets by selecting informative frames from multi-camera setups. Experimental results demonstrate that our approach consistently improves rendering quality and semantic segmentation performance, outperforming baseline methods based on random selection and uncertainty-based heuristics.