Multimodal Fusion of Skeleton Dynamics and Clinical Gait Features for Video-Based Cerebral Palsy Severity Assessment

TL;DR

This paper introduces a multimodal fusion framework combining skeleton dynamics and gait features for improved video-based cerebral palsy severity assessment, enhancing interpretability and classification accuracy.

Contribution

It presents a novel dual-stream architecture with contribution-guided gait features and skeleton dynamics fusion, improving CP severity classification performance.

Findings

Achieved 70.86% accuracy in CP severity classification

Outperformed baseline by 5.6 percentage points

Enhanced interpretability through Grad-CAM analysis

Abstract

Video-based gait analysis has become a promising approach for assessing motor impairment in children with cerebral palsy (CP). However, existing methods usually rely on either pose sequences or handcrafted gait features alone, making it difficult to simultaneously capture spatiotemporal motion patterns and clinically meaningful biomechanical information. To address this gap, we propose a multimodal fusion framework that integrates skeleton dynamics with contribution-guided clinically meaningful gait features. First, Grad-CAM analysis on a pre-trained ST-GCN backbone identified the most discriminative body keypoints, providing an interpretable basis for subsequent gait feature extraction. We then build a dual-stream architecture, with one stream modeling skeleton dynamics using ST-GCN and the other encoding gait geatures derived from the identified keypoints. By fusing the two streams through feature cross-attention improved four-level CP motor severity classification to 70.86%, outperforming the baseline by 5.6 percentage points. Overall, this work suggests that integrating skeleton dynamics with clinically meaningful gait descriptors can improve both prediction performance and biomechanical interpretability for video-based CP severity assessment.