HandCept: A Visual-Inertial Fusion Framework for Accurate Proprioception in Dexterous Hands

TL;DR

HandCept introduces a real-time visual-inertial fusion framework using a wrist-mounted camera and IMUs, achieving accurate, drift-free joint angle estimation in dexterous robotic hands, enhancing manipulation capabilities.

Contribution

The paper presents a novel zero-shot learning visual-inertial fusion framework with a latency-free EKF for accurate proprioception in dexterous hands, including a high-fidelity rendering pipeline for sim-to-real transfer.

Findings

Achieves joint angle errors between 2° and 4° without drift.

Outperforms visual-only and inertial-only methods.

Provides a stable, calibrated IMU system with a common base frame.

Abstract

As robotics progresses toward general manipulation, dexterous hands are becoming increasingly critical. However, proprioception in dexterous hands remains a bottleneck due to limitations in volume and generality. In this work, we present HandCept, a novel visual-inertial proprioception framework designed to overcome the challenges of traditional joint angle estimation methods. HandCept addresses the difficulty of achieving accurate and robust joint angle estimation in dynamic environments where both visual and inertial measurements are prone to noise and drift. It leverages a zero-shot learning approach using a wrist-mounted RGB-D camera and 9-axis IMUs, fused in real time via a latency-free Extended Kalman Filter (EKF). Our results show that HandCept achieves joint angle estimation errors between $2^{\circ}$ and $4^{\circ}$ without observable drift, outperforming visual-only and inertial-only methods. Furthermore, we validate the stability and uniformity of the IMU system, demonstrating that a common base frame across IMUs simplifies system calibration. To support sim-to-real transfer, we also open-sourced our high-fidelity rendering pipeline, which is essential for training without real-world ground truth. This work offers a robust, generalizable solution for proprioception in dexterous hands, with significant implications for robotic manipulation and human-robot interaction.