MonoMobility: Zero-Shot 3D Mobility Analysis from Monocular Videos

TL;DR

MonoMobility introduces a zero-shot framework for analyzing 3D motion parts and attributes from monocular videos without requiring annotated training data, advancing embodied intelligence capabilities.

Contribution

It presents a novel end-to-end method combining scene geometry, motion analysis, and dynamic scene optimization for zero-shot 3D mobility analysis from monocular videos.

Findings

Effective analysis of articulated object motions in real-world scenarios

Handles complex movements like rotation and translation

Operates without annotated training data

Abstract

Accurately analyzing the motion parts and their motion attributes in dynamic environments is crucial for advancing key areas such as embodied intelligence. Addressing the limitations of existing methods that rely on dense multi-view images or detailed part-level annotations, we propose an innovative framework that can analyze 3D mobility from monocular videos in a zero-shot manner. This framework can precisely parse motion parts and motion attributes only using a monocular video, completely eliminating the need for annotated training data. Specifically, our method first constructs the scene geometry and roughly analyzes the motion parts and their initial motion attributes combining depth estimation, optical flow analysis and point cloud registration method, then employs 2D Gaussian splatting for scene representation. Building on this, we introduce an end-to-end dynamic scene optimization algorithm specifically designed for articulated objects, refining the initial analysis results to ensure the system can handle 'rotation', 'translation', and even complex movements ('rotation+translation'), demonstrating high flexibility and versatility. To validate the robustness and wide applicability of our method, we created a comprehensive dataset comprising both simulated and real-world scenarios. Experimental results show that our framework can effectively analyze articulated object motions in an annotation-free manner, showcasing its significant potential in future embodied intelligence applications.