A Distance-Geometric Method for Recovering Robot Joint Angles From an RGB Image

TL;DR

This paper introduces a novel distance-geometric approach to recover robot joint angles solely from a single RGB image, enabling system recovery when traditional sensors fail, by leveraging a neural network and kinematic models.

Contribution

The paper presents a new method combining distance geometry and neural networks to estimate joint angles from RGB images, bypassing the need for proprioceptive sensors.

Findings

Accurate joint angle recovery from RGB images demonstrated on a real robot.

Method generalizes well to unseen configurations.

Combining with refinement improves accuracy.

Abstract

Autonomous manipulation systems operating in domains where human intervention is difficult or impossible (e.g., underwater, extraterrestrial or hazardous environments) require a high degree of robustness to sensing and communication failures. Crucially, motion planning and control algorithms require a stream of accurate joint angle data provided by joint encoders, the failure of which may result in an unrecoverable loss of functionality. In this paper, we present a novel method for retrieving the joint angles of a robot manipulator using only a single RGB image of its current configuration, opening up an avenue for recovering system functionality when conventional proprioceptive sensing is unavailable. Our approach, based on a distance-geometric representation of the configuration space, exploits the knowledge of a robot's kinematic model with the goal of training a shallow neural network that performs a 2D-to-3D regression of distances associated with detected structural keypoints. It is shown that the resulting Euclidean distance matrix uniquely corresponds to the observed configuration, where joint angles can be recovered via multidimensional scaling and a simple inverse kinematics procedure. We evaluate the performance of our approach on real RGB images of a Franka Emika Panda manipulator, showing that the proposed method is efficient and exhibits solid generalization ability. Furthermore, we show that our method can be easily combined with a dense refinement technique to obtain superior results.