VidSole: A Multimodal Dataset for Joint Kinetics Quantification and Disease Detection with Deep Learning

TL;DR

This paper introduces VidSole, a comprehensive multimodal dataset and deep learning methods for cost-effective, large-scale analysis of joint loading and disease risk, specifically targeting gait-related conditions like knee osteoarthritis.

Contribution

The paper presents novel instrumented insoles, a large multimodal dataset, and a deep learning pipeline for joint loading prediction, advancing biomechanical analysis outside lab environments.

Findings

Achieved 99.02% activity classification accuracy.

Estimated knee adduction moment with less than 0.5% MAE.

Demonstrated potential for clinical application in disease detection.

Abstract

Understanding internal joint loading is critical for diagnosing gait-related diseases such as knee osteoarthritis; however, current methods of measuring joint risk factors are time-consuming, expensive, and restricted to lab settings. In this paper, we enable the large-scale, cost-effective biomechanical analysis of joint loading via three key contributions: the development and deployment of novel instrumented insoles, the creation of a large multimodal biomechanics dataset (VidSole), and a baseline deep learning pipeline to predict internal joint loading factors. Our novel instrumented insole measures the tri-axial forces and moments across five high-pressure points under the foot. VidSole consists of the forces and moments measured by these insoles along with corresponding RGB video from two viewpoints, 3D body motion capture, and force plate data for over 2,600 trials of 52 diverse participants performing four fundamental activities of daily living (sit-to-stand, stand-to-sit, walking, and running). We feed the insole data and kinematic parameters extractable from video (i.e., pose, knee angle) into a deep learning pipeline consisting of an ensemble Gated Recurrent Unit (GRU) activity classifier followed by activity-specific Long Short Term Memory (LSTM) regression networks to estimate knee adduction moment (KAM), a biomechanical risk factor for knee osteoarthritis. The successful classification of activities at an accuracy of 99.02 percent and KAM estimation with mean absolute error (MAE) less than 0.5 percent*body weight*height, the current threshold for accurately detecting knee osteoarthritis with KAM, illustrates the usefulness of our dataset for future research and clinical settings.