iTACO: Interactable Digital Twins of Articulated Objects from Casually Captured RGBD Videos

TL;DR

iTACO is a novel framework that creates interactable digital twins of articulated objects from casually captured RGBD videos, enabling scalable and practical digitization for robotics and AI applications.

Contribution

The paper introduces iTACO, a coarse-to-fine method for segmenting and analyzing articulated objects from casual RGBD videos, and provides a large new dataset for evaluation.

Findings

iTACO outperforms existing methods on synthetic and real videos.

The dataset contains 784 videos of 284 objects, 20 times larger than prior datasets.

iTACO effectively handles object and camera motion, occlusions, and casual capture conditions.

Abstract

Articulated objects are prevalent in daily life. Interactable digital twins of such objects have numerous applications in embodied AI and robotics. Unfortunately, current methods to digitize articulated real-world objects require carefully captured data, preventing practical, scalable, and generalizable acquisition. We focus on motion analysis and part-level segmentation of an articulated object from a casually captured RGBD video shot with a hand-held camera. A casually captured video of an interaction with an articulated object is easy to obtain at scale using smartphones. However, this setting is challenging due to simultaneous object and camera motion and significant occlusions as the person interacts with the object. To tackle these challenges, we introduce iTACO: a coarse-to-fine framework that infers joint parameters and segments movable parts of the object from a dynamic RGBD video. To evaluate our method under this new setting, we build a dataset of 784 videos containing 284 objects across 11 categories that is 20$\times$ larger than available in prior work. We then compare our approach with existing methods that also take video as input. Our experiments show that iTACO outperforms existing articulated object digital twin methods on both synthetic and real casually captured RGBD videos.