3D Foundation Models Enable Simultaneous Geometry and Pose Estimation of Grasped Objects

TL;DR

This paper introduces a method using 3D foundation models to jointly estimate the geometry and pose of grasped objects from RGB images, enabling robots to interact more effectively with objects without camera calibration.

Contribution

The approach leverages large pre-trained 3D models and a coordinate-alignment technique to accurately estimate object geometry and pose in the robot's frame, without requiring camera extrinsic calibration.

Findings

Effective geometry and pose estimation on real-world objects

No need for camera calibration or extrinsic parameters

Enables robot manipulation based on object coordinates

Abstract

Humans have the remarkable ability to use held objects as tools to interact with their environment. For this to occur, humans internally estimate how hand movements affect the object's movement. We wish to endow robots with this capability. We contribute methodology to jointly estimate the geometry and pose of objects grasped by a robot, from RGB images captured by an external camera. Notably, our method transforms the estimated geometry into the robot's coordinate frame, while not requiring the extrinsic parameters of the external camera to be calibrated. Our approach leverages 3D foundation models, large models pre-trained on huge datasets for 3D vision tasks, to produce initial estimates of the in-hand object. These initial estimations do not have physically correct scales and are in the camera's frame. Then, we formulate, and efficiently solve, a coordinate-alignment problem to recover accurate scales, along with a transformation of the objects to the coordinate frame of the robot. Forward kinematics mappings can subsequently be defined from the manipulator's joint angles to specified points on the object. These mappings enable the estimation of points on the held object at arbitrary configurations, enabling robot motion to be designed with respect to coordinates on the grasped objects. We empirically evaluate our approach on a robot manipulator holding a diverse set of real-world objects.