Explainable Parkinsons Disease Gait Recognition Using Multimodal RGB-D Fusion and Large Language Models

TL;DR

This paper introduces an explainable multimodal RGB-D gait recognition system for Parkinson's disease that combines visual feature extraction with language models to provide clinically interpretable results, improving accuracy and robustness.

Contribution

It presents a novel multimodal framework integrating RGB-D data with large language models for explainable Parkinsonian gait analysis, enhancing interpretability and robustness.

Findings

Higher recognition accuracy compared to single-modality baselines

Improved robustness to lighting and occlusion variations

Effective translation of visual features into clinical explanations

Abstract

Accurate and interpretable gait analysis plays a crucial role in the early detection of Parkinsons disease (PD),yet most existing approaches remain limited by single-modality inputs, low robustness, and a lack of clinical transparency. This paper presents an explainable multimodal framework that integrates RGB and Depth (RGB-D) data to recognize Parkinsonian gait patterns under realistic conditions. The proposed system employs dual YOLOv11-based encoders for modality-specific feature extraction, followed by a Multi-Scale Local-Global Extraction (MLGE) module and a Cross-Spatial Neck Fusion mechanism to enhance spatial-temporal representation. This design captures both fine-grained limb motion (e.g., reduced arm swing) and overall gait dynamics (e.g., short stride or turning difficulty), even in challenging scenarios such as low lighting or occlusion caused by clothing. To ensure interpretability, a frozen Large Language Model (LLM) is incorporated to translate fused visual embeddings and structured metadata into clinically meaningful textual explanations. Experimental evaluations on multimodal gait datasets demonstrate that the proposed RGB-D fusion framework achieves higher recognition accuracy, improved robustness to environmental variations, and clear visual-linguistic reasoning compared with single-input baselines. By combining multimodal feature learning with language-based interpretability, this study bridges the gap between visual recognition and clinical understanding, offering a novel vision-language paradigm for reliable and explainable Parkinsons disease gait analysis. Code:https://github.com/manaralnaasan/RGB-D_parkinson-LLM