Detecting Activities of Daily Living in Egocentric Video to Contextualize Hand Use at Home in Outpatient Neurorehabilitation Settings

TL;DR

This study introduces an object-centric method using egocentric video and machine learning to recognize daily activities and object interactions in neurorehabilitation, providing clinically interpretable insights into hand use post-injury.

Contribution

It presents a novel object-focused approach leveraging pre-trained models to accurately detect activities of daily living in real-world rehabilitation settings, robust across impairments.

Findings

Achieved a mean weighted F1-score of 0.78 on a complex dataset

Maintained F1-score > 0.5 for all participants with leave-one-subject-out validation

Generated clinically interpretable information about object use during activities

Abstract

Wearable egocentric cameras and machine learning have the potential to provide clinicians with a more nuanced understanding of patient hand use at home after stroke and spinal cord injury (SCI). However, they require detailed contextual information (i.e., activities and object interactions) to effectively interpret metrics and meaningfully guide therapy planning. We demonstrate that an object-centric approach, focusing on what objects patients interact with rather than how they move, can effectively recognize Activities of Daily Living (ADL) in real-world rehabilitation settings. We evaluated our models on a complex dataset collected in the wild comprising 2261 minutes of egocentric video from 16 participants with impaired hand function. By leveraging pre-trained object detection and hand-object interaction models, our system achieves robust performance across different impairment levels and environments, with our best model achieving a mean weighted F1-score of 0.78 +/- 0.12 and maintaining an F1-score > 0.5 for all participants using leave-one-subject-out cross validation. Through qualitative analysis, we observe that this approach generates clinically interpretable information about functional object use while being robust to patient-specific movement variations, making it particularly suitable for rehabilitation contexts with prevalent upper limb impairment.