Lost & Found: Tracking Changes from Egocentric Observations in 3D Dynamic Scene Graphs

TL;DR

This paper introduces Lost & Found, a novel egocentric approach for tracking and updating dynamic 3D scene graphs to capture object movements and interactions, enhancing robotic manipulation and scene understanding.

Contribution

It presents a new method for online 6DoF object tracking from egocentric data, outperforming existing trackers and enabling dynamic scene graph updates for robotic tasks.

Findings

Outperforms state-of-the-art trackers by 34% and 56% in translational and rotational accuracy.

Produces smoother and more reliable 6DoF object trajectories.

Enables robotic applications like teach & repeat and object retrieval in cluttered environments.

Abstract

Recent approaches have successfully focused on the segmentation of static reconstructions, thereby equipping downstream applications with semantic 3D understanding. However, the world in which we live is dynamic, characterized by numerous interactions between the environment and humans or robotic agents. Static semantic maps are unable to capture this information, and the naive solution of rescanning the environment after every change is both costly and ineffective in tracking e.g. objects being stored away in drawers. With Lost & Found we present an approach that addresses this limitation. Based solely on egocentric recordings with corresponding hand position and camera pose estimates, we are able to track the 6DoF poses of the moving object within the detected interaction interval. These changes are applied online to a transformable scene graph that captures object-level relations. Compared to state-of-the-art object pose trackers, our approach is more reliable in handling the challenging egocentric viewpoint and the lack of depth information. It outperforms the second-best approach by 34% and 56% for translational and orientational error, respectively, and produces visibly smoother 6DoF object trajectories. In addition, we illustrate how the acquired interaction information in the dynamic scene graph can be employed in the context of robotic applications that would otherwise be unfeasible: We show how our method allows to command a mobile manipulator through teach & repeat, and how information about prior interaction allows a mobile manipulator to retrieve an object hidden in a drawer. Code, videos and corresponding data are accessible at https://behretj.github.io/LostAndFound.