Instance-Agnostic Geometry and Contact Dynamics Learning

TL;DR

This paper introduces an instance-agnostic framework that combines vision and dynamics to learn object shape, pose, and physical properties directly from RGBD videos without prior shape models.

Contribution

It presents a novel cyclic training pipeline integrating vision and dynamics modules, enabling shape and dynamic property learning without shape priors.

Findings

Successfully learns geometry and dynamics of rigid, convex objects

Improves object tracking accuracy over existing methods

Operates without category-level or instance-level shape priors

Abstract

This work presents an instance-agnostic learning framework that fuses vision with dynamics to simultaneously learn shape, pose trajectories, and physical properties via the use of geometry as a shared representation. Unlike many contact learning approaches that assume motion capture input and a known shape prior for the collision model, our proposed framework learns an object's geometric and dynamic properties from RGBD video, without requiring either category-level or instance-level shape priors. We integrate a vision system, BundleSDF, with a dynamics system, ContactNets, and propose a cyclic training pipeline to use the output from the dynamics module to refine the poses and the geometry from the vision module, using perspective reprojection. Experiments demonstrate our framework's ability to learn the geometry and dynamics of rigid and convex objects and improve upon the current tracking framework.