REST-HANDS: Rehabilitation with Egocentric Vision Using Smartglasses for Treatment of Hands after Surviving Stroke

TL;DR

This paper introduces REST-HANDS, a novel dataset and approach using egocentric smart glasses recordings to enable remote hand rehabilitation for stroke survivors, achieving high accuracy in exercise recognition and evaluation.

Contribution

It presents the first dataset of egocentric hand exercises and benchmarks for remote rehabilitation using smart glasses, demonstrating feasibility and high accuracy in key tasks.

Findings

Exercise recognition accuracy of 98.55%

Form evaluation accuracy of 86.98%

Repetition counting with mean absolute error of 1.33

Abstract

Stroke represents the third cause of death and disability worldwide, and is recognised as a significant global health problem. A major challenge for stroke survivors is persistent hand dysfunction, which severely affects the ability to perform daily activities and the overall quality of life. In order to regain their functional hand ability, stroke survivors need rehabilitation therapy. However, traditional rehabilitation requires continuous medical support, creating dependency on an overburdened healthcare system. In this paper, we explore the use of egocentric recordings from commercially available smart glasses, specifically RayBan Stories, for remote hand rehabilitation. Our approach includes offline experiments to evaluate the potential of smart glasses for automatic exercise recognition, exercise form evaluation and repetition counting. We present REST-HANDS, the first dataset of egocentric hand exercise videos. Using state-of-the-art methods, we establish benchmarks with high accuracy rates for exercise recognition (98.55%), form evaluation (86.98%), and repetition counting (mean absolute error of 1.33). Our study demonstrates the feasibility of using egocentric video from smart glasses for remote rehabilitation, paving the way for further research.