Recognizing Hand Use and Hand Role at Home After Stroke from Egocentric Video

TL;DR

This study demonstrates the feasibility of using egocentric video and AI to classify hand use in stroke survivors at home, with the Hand Object Detector outperforming other models in detecting hand activity.

Contribution

Introduces AI-based methods, including neural networks, for classifying hand use and roles from egocentric videos in a home setting after stroke.

Findings

Hand Object Detector outperformed other models in hand use detection.

Macro average MCCs were 0.50 for more-affected and 0.58 for less-affected hands.

Hand role classification performance was near zero MCC, indicating challenges in current methods.

Abstract

Introduction: Hand function is a central determinant of independence after stroke. Measuring hand use in the home environment is necessary to evaluate the impact of new interventions, and calls for novel wearable technologies. Egocentric video can capture hand-object interactions in context, as well as show how more-affected hands are used during bilateral tasks (for stabilization or manipulation). Automated methods are required to extract this information. Objective: To use artificial intelligence-based computer vision to classify hand use and hand role from egocentric videos recorded at home after stroke. Methods: Twenty-one stroke survivors participated in the study. A random forest classifier, a SlowFast neural network, and the Hand Object Detector neural network were applied to identify hand use and hand role at home. Leave-One-Subject-Out-Cross-Validation (LOSOCV) was used to evaluate the performance of the three models. Between-group differences of the models were calculated based on the Mathews correlation coefficient (MCC). Results: For hand use detection, the Hand Object Detector had significantly higher performance than the other models. The macro average MCCs using this model in the LOSOCV were 0.50 +- 0.23 for the more-affected hands and 0.58 +- 0.18 for the less-affected hands. Hand role classification had macro average MCCs in the LOSOCV that were close to zero for all models. Conclusion: Using egocentric video to capture the hand use of stroke survivors at home is feasible. Pose estimation to track finger movements may be beneficial to classifying hand roles in the future.