Sim2real Image Translation Enables Viewpoint-Robust Policies from Fixed-Camera Datasets

TL;DR

This paper introduces MANGO, a novel unpaired image translation method that enhances viewpoint robustness in vision-based robot manipulation policies by effectively translating simulated images to real-world viewpoints with minimal real data.

Contribution

MANGO employs a segmentation-conditioned InfoNCE loss and a regularized discriminator to maintain viewpoint consistency during sim2real translation, improving policy robustness.

Findings

MANGO outperforms other image translation methods in diverse viewpoint translation.

Augmentation with MANGO increases success rates in real-world manipulation tasks by over 40%.

The method requires only a small amount of real-world fixed-camera data.

Abstract

Vision-based policies for robot manipulation have achieved significant recent success, but are still brittle to distribution shifts such as camera viewpoint variations. Robot demonstration data is scarce and often lacks appropriate variation in camera viewpoints. Simulation offers a way to collect robot demonstrations at scale with comprehensive coverage of different viewpoints, but presents a visual sim2real challenge. To bridge this gap, we propose MANGO -- an unpaired image translation method with a novel segmentation-conditioned InfoNCE loss, a highly-regularized discriminator design, and a modified PatchNCE loss. We find that these elements are crucial for maintaining viewpoint consistency during sim2real translation. When training MANGO, we only require a small amount of fixed-camera data from the real world, but show that our method can generate diverse unseen viewpoints by translating simulated observations. In this setting, MANGO outperforms all other image translation methods we tested. In certain real-world tabletop manipulation tasks, MANGO augmentation increases shifted-view success rates by over 40 percentage points compared to policies trained without augmentation.