SplArt: Articulation Estimation and Part-Level Reconstruction with 3D Gaussian Splatting

TL;DR

SplArt is a self-supervised framework that uses 3D Gaussian Splatting to accurately reconstruct and infer articulation of objects from RGB images, enabling real-time rendering without needing 3D annotations.

Contribution

It introduces a category-agnostic, self-supervised method with a multi-stage optimization for articulated object reconstruction and kinematic inference using 3D Gaussian Splatting.

Findings

Achieves state-of-the-art accuracy in articulated object reconstruction.

Operates in real-time with photorealistic rendering.

Does not require 3D supervision or category-specific priors.

Abstract

Reconstructing articulated objects prevalent in daily environments is crucial for applications in augmented/virtual reality and robotics. However, existing methods face scalability limitations (requiring 3D supervision or costly annotations), robustness issues (being susceptible to local optima), and rendering shortcomings (lacking speed or photorealism). We introduce SplArt, a self-supervised, category-agnostic framework that leverages 3D Gaussian Splatting (3DGS) to reconstruct articulated objects and infer kinematics from two sets of posed RGB images captured at different articulation states, enabling real-time photorealistic rendering for novel viewpoints and articulations. SplArt augments 3DGS with a differentiable mobility parameter per Gaussian, achieving refined part segmentation. A multi-stage optimization strategy is employed to progressively handle reconstruction, part segmentation, and articulation estimation, significantly enhancing robustness and accuracy. SplArt exploits geometric self-supervision, effectively addressing challenging scenarios without requiring 3D annotations or category-specific priors. Evaluations on established and newly proposed benchmarks, along with applications to real-world scenarios using a handheld RGB camera, demonstrate SplArt's state-of-the-art performance and real-world practicality. Code is publicly available at https://github.com/ripl/splart.