Fusing Biomechanical and Spatio-Temporal Features for Fall Prediction: Characterizing and Mitigating the Simulation-to-Reality Gap

TL;DR

This paper introduces BioST-GCN, a dual-stream graph convolutional network that fuses biomechanical and spatio-temporal features for fall prediction, demonstrating improved performance on simulated datasets but highlighting challenges in real-world generalization.

Contribution

The study presents a novel BioST-GCN model combining pose and biomechanical data with cross-attention, and discusses strategies to mitigate the simulation-to-reality gap in fall prediction.

Findings

BioST-GCN outperforms baseline by 5.32% and 2.91% F1-score on simulated datasets.

Spatio-temporal attention provides interpretability of critical joints and phases.

Zero-shot generalization to unseen subjects drops to 35.9%.

Abstract

Falls are a leading cause of injury and loss of independence among older adults. Vision-based fall prediction systems offer a non-invasive solution to anticipate falls seconds before impact, but their development is hindered by the scarcity of available fall data. Contributing to these efforts, this study proposes the Biomechanical Spatio-Temporal Graph Convolutional Network (BioST-GCN), a dual-stream model that combines both pose and biomechanical information using a cross-attention fusion mechanism. Our model outperforms the vanilla ST-GCN baseline by 5.32% and 2.91% F1-score on the simulated MCF-UA stunt-actor and MUVIM datasets, respectively. The spatio-temporal attention mechanisms in the ST-GCN stream also provide interpretability by identifying critical joints and temporal phases. However, a critical simulation-reality gap persists. While our model achieves an 89.0% F1-score with full supervision on simulated data, zero-shot generalization to unseen subjects drops to 35.9%. This performance decline is likely due to biases in simulated data, such as 'intent-to-fall' cues. For older adults, particularly those with diabetes or frailty, this gap is exacerbated by their unique kinematic profiles. To address this, we propose personalization strategies and advocate for privacy-preserving data pipelines to enable real-world validation. Our findings underscore the urgent need to bridge the gap between simulated and real-world data to develop effective fall prediction systems for vulnerable elderly populations.