ArtGS:3D Gaussian Splatting for Interactive Visual-Physical Modeling and Manipulation of Articulated Objects

TL;DR

ArtGS introduces a novel framework that combines 3D Gaussian Splatting with visual-physical reasoning to improve articulated object understanding and manipulation in robotics, demonstrating superior accuracy and success rates.

Contribution

The paper presents ArtGS, a new approach integrating 3D Gaussian Splatting with visual-physical modeling for scalable, generalizable articulated object manipulation.

Findings

Outperforms previous methods in joint estimation accuracy

Achieves higher manipulation success rates in diverse environments

Demonstrates effectiveness in both simulation and real-world tests

Abstract

Articulated object manipulation remains a critical challenge in robotics due to the complex kinematic constraints and the limited physical reasoning of existing methods. In this work, we introduce ArtGS, a novel framework that extends 3D Gaussian Splatting (3DGS) by integrating visual-physical modeling for articulated object understanding and interaction. ArtGS begins with multi-view RGB-D reconstruction, followed by reasoning with a vision-language model (VLM) to extract semantic and structural information, particularly the articulated bones. Through dynamic, differentiable 3DGS-based rendering, ArtGS optimizes the parameters of the articulated bones, ensuring physically consistent motion constraints and enhancing the manipulation policy. By leveraging dynamic Gaussian splatting, cross-embodiment adaptability, and closed-loop optimization, ArtGS establishes a new framework for efficient, scalable, and generalizable articulated object modeling and manipulation. Experiments conducted in both simulation and real-world environments demonstrate that ArtGS significantly outperforms previous methods in joint estimation accuracy and manipulation success rates across a variety of articulated objects. Additional images and videos are available on the project website: https://sites.google.com/view/artgs/home