Monocular Biomechanical Tracking of Fingers with Inverse Kinematics to Foundation Models

TL;DR

This paper introduces a novel monocular video method combining foundation models and inverse kinematics to accurately track finger biomechanics, enabling detailed hand movement analysis from single-view videos.

Contribution

It integrates SAM 3D Body with biomechanical modeling and GPU acceleration to estimate finger joint angles from monocular video, a significant advancement over existing methods.

Findings

Achieved approximately 10 degrees finger joint angle accuracy.

Hand position errors around 6 mm after alignment.

Consistent results across different camera viewpoints.

Abstract

Accurate hand and finger tracking from video has significant clinical applications for monitoring activities of daily living and measuring range of motion, yet monocular video approaches for obtaining hand biomechanics remain under-developed. We present a method that combines the SAM 3D Body foundation model with inverse kinematics optimization in a full-body biomechanical model to extract anatomically-constrained finger joint angles from single-view video. We port SAM 3D Body from PyTorch to JAX for integration with MuJoCo-MJX, enabling GPU-accelerated optimization, and develop a novel mapping between the Momentum Human Rig (MHR) outputs and biomechanical model markers. Validation against 8-camera multiview reconstruction on 4,590 frames from 7 participants performing a variety of hand poses and object manipulation tasks shows finger joint angle errors of approximately 10 degrees and hand position errors of approximately 6 mm, after Procrustes alignment. Results were consistent across camera viewpoints and robust to different methods for producing reference values from multiview video. This work extends monocular biomechanical analysis to detailed finger tracking, expanding access to quantitative characterization of hand movement from readily available video.