EasyHeC++: Fully Automatic Hand-Eye Calibration with Pretrained Image Models

TL;DR

EasyHeC++ introduces a fully automatic, marker-free hand-eye calibration system that leverages pretrained image models and differentiable rendering, achieving high accuracy across diverse robots and camera setups without manual intervention.

Contribution

It is the first system enabling automatic, training-free hand-eye calibration for any robot arm using pretrained models and differentiable rendering techniques.

Findings

Achieves superior accuracy in synthetic and real-world tests.

Works across various robot arms and camera configurations.

Eliminates need for manual calibration or specialized markers.

Abstract

Hand-eye calibration plays a fundamental role in robotics by directly influencing the efficiency of critical operations such as manipulation and grasping. In this work, we present a novel framework, EasyHeC++, designed for fully automatic hand-eye calibration. In contrast to previous methods that necessitate manual calibration, specialized markers, or the training of arm-specific neural networks, our approach is the first system that enables accurate calibration of any robot arm in a marker-free, training-free, and fully automatic manner. Our approach employs a two-step process. First, we initialize the camera pose using a sampling or feature-matching-based method with the aid of pretrained image models. Subsequently, we perform pose optimization through differentiable rendering. Extensive experiments demonstrate the system's superior accuracy in both synthetic and real-world datasets across various robot arms and camera settings. Project page: https://ootts.github.io/easyhec_plus.