CL3R: 3D Reconstruction and Contrastive Learning for Enhanced Robotic Manipulation Representations

TL;DR

CL3R is a novel 3D pre-training framework that combines spatial and semantic learning to improve robotic manipulation perception, generalization, and robustness across viewpoints.

Contribution

It introduces a unified 3D pre-training approach integrating point cloud autoencoding and contrastive learning with multi-view fusion for better robotic perception.

Findings

Outperforms existing methods in simulation and real-world tests.

Enhances generalization across different camera viewpoints.

Improves visuomotor policy learning for robotic manipulation.

Abstract

Building a robust perception module is crucial for visuomotor policy learning. While recent methods incorporate pre-trained 2D foundation models into robotic perception modules to leverage their strong semantic understanding, they struggle to capture 3D spatial information and generalize across diverse camera viewpoints. These limitations hinder the policy's effectiveness, especially in fine-grained robotic manipulation scenarios. To address these challenges, we propose CL3R, a novel 3D pre-training framework designed to enhance robotic manipulation policies. Our method integrates both spatial awareness and semantic understanding by employing a point cloud Masked Autoencoder to learn rich 3D representations while leveraging pre-trained 2D foundation models through contrastive learning for efficient semantic knowledge transfer. Additionally, we propose a 3D visual representation pre-training framework for robotic tasks. By unifying coordinate systems across datasets and introducing random fusion of multi-view point clouds, we mitigate camera view ambiguity and improve generalization, enabling robust perception from novel viewpoints at test time. Extensive experiments in both simulation and the real world demonstrate the superiority of our method, highlighting its effectiveness in visuomotor policy learning for robotic manipulation.