Agentic Pipeline for Self-Synchronized Multiview Joint Angle Monitoring in Uncalibrated Environments

TL;DR

This paper introduces an agentic pipeline that enables self-synchronized, multi-view joint angle monitoring in uncalibrated environments, facilitating long-term rehabilitation for SCI patients without requiring calibration or hardware triggers.

Contribution

It presents a novel, fully automated system combining multimodal large language models and agent-based pose selection for accurate, self-deployed kinematic monitoring in uncalibrated settings.

Findings

Achieved an MAE of 5.97 degrees in joint angle estimation.

Demonstrated high correlation coefficient of 0.962 with Vicon system.

Validated system performance in uncalibrated, real-world environments.

Abstract

Kinematic monitoring plays a critical role in long-term rehabilitation for patients with spinal cord injury (SCI), where multi-view markerless motion capture methods have shown significant potential. However, owing to the reliance on calibration and the difficulty of achieving multi-view synchronization, their deployment in patient self-deployed environments remains challenging. In this work, we propose an agentic pipeline for self-synchronized multi-view joint angle monitoring in uncalibrated environments using two cameras without hardware triggers. The Multimodal large language models enable automatic video synchronization and agent-driven self-verification. State-of-the-art monocular 2D pose estimation models are employed to extract candidate poses, where an agent-based selection mechanism is then applied to automatically identify and track the target subject, thereby producing consistent 2D poses in the presence of multiple individuals and occlusions. Such 2D poses are optimized to estimate joint angles from uncalibrated multi-view pose sequences, ensuring interpretability through explicit geometric modeling. Validation against Vicon system demonstrated the strong performance, achieving an MAE of $5.97^\circ \pm 2.36^\circ$ and a Pearson correlation coefficient of $0.962 \pm 0.014$. The proposed method is expected to provide a practical, patient self-deployable system to perform daily kinematic monitoring in uncalibrated home environments.