Multi-Rigid-Body Approximation of Human Hands with Application to Digital Twin

TL;DR

This paper introduces a pipeline for creating realistic, real-time simulatable multi-rigid-body models of human hands from motion capture data, suitable for digital twin applications.

Contribution

It develops a method to convert personalized hand models into kinematically consistent rigid-body representations with closed-form joint rotation projections and BCH-corrected iterative solutions.

Findings

Achieves sub-centimeter reconstruction accuracy

Enables real-time physics simulation of hand movements

Successfully reproduces human grasping behaviors

Abstract

Human hand simulation plays a critical role in digital twin applications, requiring models that balance anatomical fidelity with computational efficiency. We present a complete pipeline for constructing multi-rigid-body approximations of human hands that preserve realistic appearance while enabling real-time physics simulation. Starting from optical motion capture of a specific human hand, we construct a personalized MANO (Multi-Abstracted hand model with Neural Operations) model and convert it to a URDF (Unified Robot Description Format) representation with anatomically consistent joint axes. The key technical challenge is projecting MANO's unconstrained SO(3) joint rotations onto the kinematically constrained joints of the rigid-body model. We derive closed-form solutions for single degree-of-freedom joints and introduce a Baker-Campbell-Hausdorff (BCH)-corrected iterative method for two degree-of-freedom joints that properly handles the non-commutativity of rotations. We validate our approach through digital twin experiments where reinforcement learning policies control the multi-rigid-body hand to replay captured human demonstrations. Quantitative evaluation shows sub-centimeter reconstruction error and successful grasp execution across diverse manipulation tasks.