Object and Contact Point Tracking in Demonstrations Using 3D Gaussian Splatting

TL;DR

This paper presents a novel method that uses 3D Gaussian Splatting and FoundationPose to extract contact points and track objects in video demonstrations, improving robot learning of complex interactions.

Contribution

It introduces a new approach combining 3D Gaussian Splatting and FoundationPose to enhance object and contact point tracking in robotic imitation learning from videos.

Findings

Improved accuracy in contact point detection.

Enhanced tracking of articulated objects like doors and drawers.

Facilitated better task understanding for robots.

Abstract

This paper introduces a method to enhance Interactive Imitation Learning (IIL) by extracting touch interaction points and tracking object movement from video demonstrations. The approach extends current IIL systems by providing robots with detailed knowledge of both where and how to interact with objects, particularly complex articulated ones like doors and drawers. By leveraging cutting-edge techniques such as 3D Gaussian Splatting and FoundationPose for tracking, this method allows robots to better understand and manipulate objects in dynamic environments. The research lays the foundation for more effective task learning and execution in autonomous robotic systems.