NeuralDiff: Segmenting 3D objects that move in egocentric videos

TL;DR

NeuralDiff introduces a neural rendering approach to segment and separate static backgrounds from dynamic objects and actors in egocentric videos, effectively handling large camera viewpoint changes.

Contribution

The paper presents a novel triple-stream neural rendering network that decomposes 3D scenes into static and dynamic components in egocentric videos, outperforming existing neural rendering methods.

Findings

Successfully separates static and dynamic scene components.

Achieves accurate segmentation of moving objects in complex 3D environments.

Establishes a new benchmark for dynamic object segmentation in egocentric videos.

Abstract

Given a raw video sequence taken from a freely-moving camera, we study the problem of decomposing the observed 3D scene into a static background and a dynamic foreground containing the objects that move in the video sequence. This task is reminiscent of the classic background subtraction problem, but is significantly harder because all parts of the scene, static and dynamic, generate a large apparent motion due to the camera large viewpoint change. In particular, we consider egocentric videos and further separate the dynamic component into objects and the actor that observes and moves them. We achieve this factorization by reconstructing the video via a triple-stream neural rendering network that explains the different motions based on corresponding inductive biases. We demonstrate that our method can successfully separate the different types of motion, outperforming recent neural rendering baselines at this task, and can accurately segment moving objects. We do so by assessing the method empirically on challenging videos from the EPIC-KITCHENS dataset which we augment with appropriate annotations to create a new benchmark for the task of dynamic object segmentation on unconstrained video sequences, for complex 3D environments.