Multi-View Masked World Models for Visual Robotic Manipulation

TL;DR

This paper introduces a multi-view masked autoencoder that learns robust visual representations from multiple camera views, enabling effective robotic manipulation and policy transfer without camera calibration.

Contribution

We propose a multi-view masked autoencoder for learning representations from multi-view data, improving robotic manipulation and policy transfer in uncalibrated, viewpoint-randomized scenarios.

Findings

Effective multi-view control demonstrated

Robust policy transfer without camera calibration

Enhanced representation learning from multi-view data

Abstract

Visual robotic manipulation research and applications often use multiple cameras, or views, to better perceive the world. How else can we utilize the richness of multi-view data? In this paper, we investigate how to learn good representations with multi-view data and utilize them for visual robotic manipulation. Specifically, we train a multi-view masked autoencoder which reconstructs pixels of randomly masked viewpoints and then learn a world model operating on the representations from the autoencoder. We demonstrate the effectiveness of our method in a range of scenarios, including multi-view control and single-view control with auxiliary cameras for representation learning. We also show that the multi-view masked autoencoder trained with multiple randomized viewpoints enables training a policy with strong viewpoint randomization and transferring the policy to solve real-robot tasks without camera calibration and an adaptation procedure. Video demonstrations are available at: https://sites.google.com/view/mv-mwm.