Invariance Co-training for Robot Visual Generalization

TL;DR

This paper introduces an invariance co-training approach that enhances robot visual generalization across diverse observational conditions by leveraging both robotic demonstrations and synthetic images, significantly improving robustness.

Contribution

It proposes a novel auxiliary task framework for invariance, combining real and synthetic data to improve robot visual generalization across various environmental variations.

Findings

18% performance improvement over existing methods

Effective generalization to unseen viewpoints and lighting

Utilizes both robotic and synthetic data for training

Abstract

Reasoning from diverse observations is a fundamental capability for generalist robot policies to operate in a wide range of environments. Despite recent advancements, many large-scale robotic policies still remain sensitive to key sources of observational variation such as changes in camera perspective, lighting, and the presence of distractor objects. We posit that the limited generalizability of these models arises from the substantial diversity required to robustly cover these quasistatic axes, coupled with the current scarcity of large-scale robotic datasets that exhibit rich variation across them. In this work, we propose to systematically examine what robots need to generalize across these challenging axes by introducing two key auxiliary tasks, state similarity and invariance to observational perturbations, applied to both demonstration data and static visual data. We then show that via these auxiliary tasks, leveraging both more-expensive robotic demonstration data and less-expensive, visually rich synthetic images generated from non-physics-based simulation (for example, Unreal Engine) can lead to substantial increases in generalization to unseen camera viewpoints, lighting configurations, and distractor conditions. Our results demonstrate that co-training on this diverse data improves performance by 18 percent over existing generative augmentation methods. For more information and videos, please visit https://invariance-cotraining.github.io