Mitigating the Human-Robot Domain Discrepancy in Visual Pre-training for Robotic Manipulation

TL;DR

This paper introduces a novel adaptation method that uses paired human-robot videos and contrastive alignment to improve visual representations for robotic manipulation, significantly enhancing performance across diverse tasks.

Contribution

The paper presents a new adaptation paradigm leveraging paired human-robot videos and a contrastive loss to bridge the human-robot domain gap in visual pre-training for manipulation.

Findings

Over 7% improvement in success rate across multiple tasks

Significant gains on both simulated and real-world benchmarks

Effective in single-task and multi-task settings

Abstract

Learning generalizable visual representations across different embodied environments is essential for effective robotic manipulation in real-world scenarios. However, the limited scale and diversity of robot demonstration data pose a significant challenge. Recent research has explored leveraging large-scale human activity data for pre-training, but the substantial morphological differences between humans and robots introduce a significant human-robot domain discrepancy, hindering the generalization of these models to downstream manipulation tasks. To overcome this, we propose a novel adaptation paradigm that leverages readily available paired human-robot video data to bridge the domain gap. Our method employs a human-robot contrastive alignment loss to align the semantics of human and robot videos, adapting pre-trained models to the robot domain in a parameter-efficient manner. Experiments on 20 simulated tasks across two different benchmarks and five real-world tasks demonstrate significant improvements. These results span both single-task and language-conditioned multi-task settings, evaluated using two different pre-trained models. Compared to existing pre-trained models, our adaptation method improves the average success rate by over 7% across multiple tasks on both simulated benchmarks and real-world evaluations.