Beyond the joystick: deep learning games for hand movement recovery

TL;DR

This study shows how deep learning and games can help people recover hand movement by making rehabilitation fun and interactive.

Contribution

A novel, accessible, and affordable gesture-controlled rehabilitation system using deep learning and gamification.

Findings

Gesture recognition accuracy and game control performance were consistent and stable.

User usability scores exceeded the benchmark, showing the system is intuitive and engaging.

The system offers an effective and interactive alternative to existing low-cost rehabilitation tools.

Abstract

This research explores the use of Deep Learning (DL) techniques for hand and gesture recognition to support hand rehabilitation programs. The primary objective is to enhance cognitive function and hand-eye coordination through gamified therapeutic exercises that track and respond to hand gestures in real time.

Pre-trained Convolutional Neural Network (CNN) models were employed for hand recognition using Google's open-source MediaPipe Library. Four classic arcade games-Pong, Tetris, Fruit Ninja, and a Virtual Keyboard-were redeveloped as gesture-controlled rehabilitation tools within a web-based interface built using the Phaser.js framework. A score-based system was implemented to track user performance and progress. Usability was evaluated using the System Usability Scale (SUS), and statistical validation was performed with a one-sample t-test against the industry benchmark.

Data collected from 15 participants demonstrated consistent gesture recognition accuracy and stable game control performance. The SUS evaluation indicated favorable user responses, with usability scores exceeding the benchmark threshold, suggesting that participants found the system intuitive and engaging.

The study confirms the feasibility of using monocular camera-based computer vision for hand rehabilitation. Compared to existing low-cost rehabilitation tools, the proposed system provides an accessible, interactive, and affordable alternative. Integrating gesture recognition with gamified interfaces effectively supports dexterity recovery and motivates users through engaging gameplay. These findings establish a foundation for future development of AI-based rehabilitation platforms using standard camera devices.

Figures