Fusing uncalibrated IMUs and handheld smartphone video to reconstruct knee kinematics

TL;DR

This paper introduces a method to fuse uncalibrated smartphone video and wearable sensor data to accurately reconstruct knee movements, improving clinical gait analysis without extensive calibration.

Contribution

It presents a novel fusion approach that combines uncalibrated video and sensor data at full temporal resolution, eliminating the need for time-consuming calibration procedures.

Findings

Fusion reduces knee reconstruction error during walking.

Sensor data enables tracking during visual occlusion.

Method validated on diverse participant groups.

Abstract

Video and wearable sensor data provide complementary information about human movement. Video provides a holistic understanding of the entire body in the world while wearable sensors provide high-resolution measurements of specific body segments. A robust method to fuse these modalities and obtain biomechanically accurate kinematics would have substantial utility for clinical assessment and monitoring. While multiple video-sensor fusion methods exist, most assume that a time-intensive, and often brittle, sensor-body calibration process has already been performed. In this work, we present a method to combine handheld smartphone video and uncalibrated wearable sensor data at their full temporal resolution. Our monocular, video-only, biomechanical reconstruction already performs well, with only several degrees of error at the knee during walking compared to markerless motion capture. Reconstructing from a fusion of video and wearable sensor data further reduces this error. We validate this in a mixture of people with no gait impairments, lower limb prosthesis users, and individuals with a history of stroke. We also show that sensor data allows tracking through periods of visual occlusion.