Cloth-Splatting: 3D Cloth State Estimation from RGB Supervision

TL;DR

Cloth-Splatting is a novel method that estimates 3D cloth states from RGB images by combining a differentiable 3D mesh representation with Gaussian Splatting, enabling efficient and accurate state refinement using only RGB supervision.

Contribution

It introduces a new framework coupling 3D mesh-based representations with Gaussian Splatting for differentiable cloth state estimation from RGB images.

Findings

Improves accuracy over existing methods.

Reduces convergence time for cloth state estimation.

Effective with only RGB supervision.

Abstract

We introduce Cloth-Splatting, a method for estimating 3D states of cloth from RGB images through a prediction-update framework. Cloth-Splatting leverages an action-conditioned dynamics model for predicting future states and uses 3D Gaussian Splatting to update the predicted states. Our key insight is that coupling a 3D mesh-based representation with Gaussian Splatting allows us to define a differentiable map between the cloth state space and the image space. This enables the use of gradient-based optimization techniques to refine inaccurate state estimates using only RGB supervision. Our experiments demonstrate that Cloth-Splatting not only improves state estimation accuracy over current baselines but also reduces convergence time.