DeformGS: Scene Flow in Highly Deformable Scenes for Deformable Object Manipulation

TL;DR

DeformGS is a novel method for dense 3D scene flow estimation in highly deformable scenes, enabling improved robotic manipulation of objects like cloth by leveraging multi-view video, Gaussian splatting, and physics-inspired regularization.

Contribution

It introduces DeformGS, a new approach combining Gaussian splatting, neural-voxel encoding, and physics-based regularization for accurate scene flow in deformable objects.

Findings

DeformGS improves 3D tracking accuracy by 55.8% over previous methods.

Achieves a median tracking error of 3.3 mm on a large cloth scene.

Performs well even with shadows and occlusions in complex scenes.

Abstract

Teaching robots to fold, drape, or reposition deformable objects such as cloth will unlock a variety of automation applications. While remarkable progress has been made for rigid object manipulation, manipulating deformable objects poses unique challenges, including frequent occlusions, infinite-dimensional state spaces and complex dynamics. Just as object pose estimation and tracking have aided robots for rigid manipulation, dense 3D tracking (scene flow) of highly deformable objects will enable new applications in robotics while aiding existing approaches, such as imitation learning or creating digital twins with real2sim transfer. We propose DeformGS, an approach to recover scene flow in highly deformable scenes, using simultaneous video captures of a dynamic scene from multiple cameras. DeformGS builds on recent advances in Gaussian splatting, a method that learns the properties of a large number of Gaussians for state-of-the-art and fast novel-view synthesis. DeformGS learns a deformation function to project a set of Gaussians with canonical properties into world space. The deformation function uses a neural-voxel encoding and a multilayer perceptron (MLP) to infer Gaussian position, rotation, and a shadow scalar. We enforce physics-inspired regularization terms based on conservation of momentum and isometry, which leads to trajectories with smaller trajectory errors. We also leverage existing foundation models SAM and XMEM to produce noisy masks, and learn a per-Gaussian mask for better physics-inspired regularization. DeformGS achieves high-quality 3D tracking on highly deformable scenes with shadows and occlusions. In experiments, DeformGS improves 3D tracking by an average of 55.8% compared to the state-of-the-art. With sufficient texture, DeformGS achieves a median tracking error of 3.3 mm on a cloth of 1.5 x 1.5 m in area. Website: https://deformgs.github.io