Bi-Manual Joint Camera Calibration and Scene Representation

TL;DR

Bi-JCR is a novel framework that enables joint calibration of multiple robot-mounted cameras and scene representation without traditional marker-based methods, using 3D foundation models and RGB images.

Contribution

It introduces a marker-free, joint calibration and scene representation method for multi-manipulator robots using dense 3D models and RGB images.

Findings

Robust calibration across various tabletop environments.

Unified 3D scene representation supports downstream tasks.

Effective multi-view correspondence without calibration markers.

Abstract

Robot manipulation, especially bimanual manipulation, often requires setting up multiple cameras on multiple robot manipulators. Before robot manipulators can generate motion or even build representations of their environments, the cameras rigidly mounted to the robot need to be calibrated. Camera calibration is a cumbersome process involving collecting a set of images, with each capturing a pre-determined marker. In this work, we introduce the Bi-Manual Joint Calibration and Representation Framework (Bi-JCR). Bi-JCR enables multiple robot manipulators, each with cameras mounted, to circumvent taking images of calibration markers. By leveraging 3D foundation models for dense, marker-free multi-view correspondence, Bi-JCR jointly estimates: (i) the extrinsic transformation from each camera to its end-effector, (ii) the inter-arm relative poses between manipulators, and (iii) a unified, scale-consistent 3D representation of the shared workspace, all from the same captured RGB image sets. The representation, jointly constructed from images captured by cameras on both manipulators, lives in a common coordinate frame and supports collision checking and semantic segmentation to facilitate downstream bimanual coordination tasks. We empirically evaluate the robustness of Bi-JCR on a variety of tabletop environments, and demonstrate its applicability on a variety of downstream tasks.