OCRA: Object-Centric Learning with 3D and Tactile Priors for Human-to-Robot Action Transfer

TL;DR

OCRA introduces an object-centric, multimodal framework combining 3D vision and tactile data to improve human-to-robot action transfer, enabling robots to learn manipulation tasks more robustly from videos.

Contribution

The paper presents a novel object-centric learning framework that integrates 3D visual and tactile priors for improved robot manipulation from human demonstrations.

Findings

OCRA outperforms existing baselines in vision-only tasks.

OCRA effectively fuses 3D and tactile data for robust manipulation.

The approach demonstrates significant improvements in real-world experiments.

Abstract

We present OCRA, an Object-Centric framework for video-based human-to-Robot Action transfer that learns directly from human demonstration videos to enable robust manipulation. Object-centric learning emphasizes task-relevant objects and their interactions while filtering out irrelevant background, providing a natural and scalable way to teach robots. OCRA leverages multi-view RGB videos, the state-of-the-art 3D foundation model VGGT, and advanced detection and segmentation models to reconstruct object-centric 3D point clouds, capturing rich interactions between objects. To handle properties not easily perceived by vision alone, we incorporate tactile priors via a large-scale dataset of over one million tactile images. These 3D and tactile priors are fused through a multimodal module (ResFiLM) and fed into a Diffusion Policy to generate robust manipulation actions. Extensive experiments on both vision-only and visuo-tactile tasks show that OCRA significantly outperforms existing baselines and ablations, demonstrating its effectiveness for learning from human demonstration videos.