ExeChecker: Where Did I Go Wrong?

TL;DR

ExeChecker is a contrastive learning framework that interprets rehabilitation exercises by identifying incorrect movements and highlighting relevant joints, improving feedback accuracy for users during exercise performance.

Contribution

The paper introduces ExeChecker, a novel contrastive learning approach that leverages human pose estimation and graph-attention networks for exercise interpretation and error localization.

Findings

Outperforms baseline in identifying relevant joints

Effective on in-house and public datasets

Provides informative feedback for rehabilitation exercises

Abstract

In this paper, we present a contrastive learning based framework, ExeChecker, for the interpretation of rehabilitation exercises. Our work builds upon state-of-the-art advances in the area of human pose estimation, graph-attention neural networks, and transformer interpretablity. The downstream task is to assist rehabilitation by providing informative feedback to users while they are performing prescribed exercises. We utilize a contrastive learning strategy during training. Given a tuple of correctly and incorrectly executed exercises, our model is able to identify and highlight those joints that are involved in an incorrect movement and thus require the user's attention. We collected an in-house dataset, ExeCheck, with paired recordings of both correct and incorrect execution of exercises. In our experiments, we tested our method on this dataset as well as the UI-PRMD dataset and found ExeCheck outperformed the baseline method using pairwise sequence alignment in identifying joints of physical relevance in rehabilitation exercises.