Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-aware Spatio-Temporal Sampling

TL;DR

Gear-NeRF enhances dynamic scene rendering by integrating semantic segmentation and motion-aware stratified sampling, enabling high-quality free-viewpoint synthesis and object tracking with improved efficiency and scene understanding.

Contribution

It introduces a novel semantic and motion-aware stratified sampling framework for NeRFs, enabling better dynamic scene modeling and free-viewpoint object tracking.

Findings

Achieves state-of-the-art rendering quality on multiple datasets.

Enables free-viewpoint tracking of objects in dynamic scenes.

Improves reconstruction quality with limited computational resources.

Abstract

Extensions of Neural Radiance Fields (NeRFs) to model dynamic scenes have enabled their near photo-realistic, free-viewpoint rendering. Although these methods have shown some potential in creating immersive experiences, two drawbacks limit their ubiquity: (i) a significant reduction in reconstruction quality when the computing budget is limited, and (ii) a lack of semantic understanding of the underlying scenes. To address these issues, we introduce Gear-NeRF, which leverages semantic information from powerful image segmentation models. Our approach presents a principled way for learning a spatio-temporal (4D) semantic embedding, based on which we introduce the concept of gears to allow for stratified modeling of dynamic regions of the scene based on the extent of their motion. Such differentiation allows us to adjust the spatio-temporal sampling resolution for each region in proportion to its motion scale, achieving more photo-realistic dynamic novel view synthesis. At the same time, almost for free, our approach enables free-viewpoint tracking of objects of interest - a functionality not yet achieved by existing NeRF-based methods. Empirical studies validate the effectiveness of our method, where we achieve state-of-the-art rendering and tracking performance on multiple challenging datasets.