Semantic Flow: Learning Semantic Field of Dynamic Scenes from Monocular Videos

TL;DR

Semantic Flow introduces a neural approach to learn semantic representations of dynamic scenes from monocular videos by modeling 3D motion flows, enabling advanced scene understanding and editing tasks.

Contribution

It presents a novel method that learns semantic 3D flows from monocular videos, addressing 2D-3D ambiguity and enabling new dynamic scene applications.

Findings

Successfully learns from multiple dynamic scenes

Supports instance-level scene editing and semantic completion

Enables dynamic scene tracking and semantic adaptation

Abstract

In this work, we pioneer Semantic Flow, a neural semantic representation of dynamic scenes from monocular videos. In contrast to previous NeRF methods that reconstruct dynamic scenes from the colors and volume densities of individual points, Semantic Flow learns semantics from continuous flows that contain rich 3D motion information. As there is 2D-to-3D ambiguity problem in the viewing direction when extracting 3D flow features from 2D video frames, we consider the volume densities as opacity priors that describe the contributions of flow features to the semantics on the frames. More specifically, we first learn a flow network to predict flows in the dynamic scene, and propose a flow feature aggregation module to extract flow features from video frames. Then, we propose a flow attention module to extract motion information from flow features, which is followed by a semantic network to output semantic logits of flows. We integrate the logits with volume densities in the viewing direction to supervise the flow features with semantic labels on video frames. Experimental results show that our model is able to learn from multiple dynamic scenes and supports a series of new tasks such as instance-level scene editing, semantic completions, dynamic scene tracking and semantic adaption on novel scenes. Codes are available at https://github.com/tianfr/Semantic-Flow/.