SAM4Dcap: Training-free Biomechanical Twin System from Monocular Video

TL;DR

SAM4Dcap is a novel, training-free framework that estimates biomechanical metrics from monocular videos, enabling accessible motion analysis without laboratory equipment.

Contribution

It integrates 4D human mesh recovery with biomechanical simulation, providing an open-source, end-to-end solution that requires no additional training.

Findings

Achieves knee kinematic predictions comparable to multi-view systems

Demonstrates potential for home-based biomechanical analysis

Identifies some discrepancies in hip flexion and residual jitter

Abstract

Quantitative biomechanical analysis is essential for clinical diagnosis and injury prevention but is often restricted to laboratories due to the high cost of optical motion capture systems. While multi-view video approaches have lowered barriers, they remain impractical for home-based scenarios requiring monocular capture. This paper presents SAM4Dcap, an open-source, end-to-end framework for estimating biomechanical metrics from monocular video without additional training. SAM4Dcap integrates the temporally consistent 4D human mesh recovery of SAM-Body4D with the OpenSim biomechanical solver. The pipeline converts reconstructed meshes into trajectory files compatible with diverse musculoskeletal models. We introduce automated prompting strategies and a Linux-native build for processing. Preliminary evaluations on walking and drop-jump tasks indicate that SAM4Dcap has the potential to achieve knee kinematic predictions comparable to multi-view systems, although some discrepancies in hip flexion and residual jitter remain. By bridging advanced computer vision with established biomechanical simulation, SAM4Dcap provides a flexible, accessible foundation for non-laboratory motion analysis.