CloDS: Visual-Only Unsupervised Cloth Dynamics Learning in Unknown Conditions

TL;DR

This paper introduces CloDS, an unsupervised framework for learning cloth dynamics solely from multi-view visual observations, capable of handling unknown conditions and severe deformations without prior physical knowledge.

Contribution

CloDS is a novel unsupervised learning framework that models cloth dynamics from visual data using a three-stage pipeline with mesh-based Gaussian splatting and dual-position opacity modulation.

Findings

CloDS effectively learns cloth dynamics from visual data.

The method generalizes well to unseen configurations.

It handles large deformations and self-occlusions robustly.

Abstract

Deep learning has demonstrated remarkable capabilities in simulating complex dynamic systems. However, existing methods require known physical properties as supervision or inputs, limiting their applicability under unknown conditions. To explore this challenge, we introduce Cloth Dynamics Grounding (CDG), a novel scenario for unsupervised learning of cloth dynamics from multi-view visual observations. We further propose Cloth Dynamics Splatting (CloDS), an unsupervised dynamic learning framework designed for CDG. CloDS adopts a three-stage pipeline that first performs video-to-geometry grounding and then trains a dynamics model on the grounded meshes. To cope with large non-linear deformations and severe self-occlusions during grounding, we introduce a dual-position opacity modulation that supports bidirectional mapping between 2D observations and 3D geometry via mesh-based Gaussian splatting in video-to-geometry grounding stage. It jointly considers the absolute and relative position of Gaussian components. Comprehensive experimental evaluations demonstrate that CloDS effectively learns cloth dynamics from visual data while maintaining strong generalization capabilities for unseen configurations. Our code is available at https://github.com/whynot-zyl/CloDS. Visualization results are available at https://github.com/whynot-zyl/CloDS_video}.%\footnote{As in this example.