# Study on Auxiliary Rehabilitation System of Hand Function Based on Machine Learning with Visual Sensors

## TL;DR

This study develops a machine learning-based system using visual sensors to help stroke patients recover hand function through interactive rehabilitation at home.

## Contribution

The paper introduces a novel rehabilitation system combining ResNet with Focal Loss and Leap Motion 2 for accurate hand gesture recognition and assessment.

## Key findings

- The system achieved a Macro F1 score of 91.0% and validation accuracy of 90.9% for gesture recognition.
- A static assessment gesture dataset with 502,401 frames was created using the FMA scale and Leap Motion 2.
- The system demonstrates technical feasibility and high accuracy for home-based rehabilitation training.

## Abstract

This study aims to assess hand function recovery in stroke patients during the mid-to-late Brunnstrom stages and to encourage active participation in rehabilitation exercises. To this end, a deep residual network (ResNet) integrated with Focal Loss is employed for gesture recognition, achieving a Macro F1 score of 91.0% and a validation accuracy of 90.9%. Leveraging the millimetre-level precision of Leap Motion 2 hand tracking, a mapping relationship for hand skeletal joint points was established, and a static assessment gesture data set containing 502,401 frames was collected through analysis of the FMA scale. The system implements an immersive augmented reality interaction through the Unity development platform; C# algorithms were designed for real-time motion range quantification. Finally, the paper designs a rehabilitation system framework tailored for home and community environments, including system module workflows, assessment modules, and game logic. Experimental results demonstrate the technical feasibility and high accuracy of the automated system for assessment and rehabilitation training. The system is designed to support stroke patients in home and community settings, with the potential to enhance rehabilitation motivation, interactivity, and self-efficacy. This work presents an integrated research framework encompassing hand modelling and deep learning-based recognition. It offers the possibility of feasible and economical solutions for stroke survivors, laying the foundation for future clinical applications.