Unify Robot Actions in Camera Frame

TL;DR

This paper introduces CalibAll, a novel, training-free pipeline that calibrates camera extrinsics and standardizes robot actions in the camera frame across diverse platforms, enhancing cross-embodiment learning.

Contribution

CalibAll provides a universal, calibration-free method to annotate datasets with camera extrinsics and standardized actions, enabling better cross-platform robot learning.

Findings

CalibAll successfully calibrates 97K data episodes across 16 datasets and 4 robot platforms.

Using camera-frame actions improves cross-embodiment pretraining performance.

The approach outperforms existing offline calibration methods in accuracy and robustness.

Abstract

Cross-embodiment robot learning requires a unified action representation with consistent semantics across robot platforms. Existing representations suffer from platform-specific inconsistencies, while current solutions either maintain embodiment-specific action heads or learn latent action spaces, without fundamentally resolving the mismatch. We propose to unify robot actions in the camera frame using camera extrinsics, so that actions share consistent geometric semantics across different robot embodiments, including both single-arm and bimanual robots. However, most existing datasets lack camera extrinsic annotations, and existing offline calibration methods either suffer from local minima or require robot-specific training data. To address this gap, we present CalibAll, a training-free, robot-independent annotation pipeline that estimates camera extrinsics for offline datasets and converts heterogeneous robot actions into standardized camera-frame actions. CalibAll follows a coarse-to-fine calibration strategy: temporal PnP provides a stable initialization, followed by differentiable rendering-based refinement for high precision. Beyond extrinsics, CalibAll produces standardized TCP-pose actions and auxiliary annotations. We apply CalibAll to 16 datasets across 4 robot platforms, producing approximately 97K calibrated data episodes. Downstream simulation and real-robot experiments show that cross-embodiment pretraining with camera-frame actions achieves state-of-the-art performance.