Domain Knowledge-Informed Self-Supervised Representations for Workout Form Assessment

TL;DR

This paper introduces a domain knowledge-informed self-supervised learning approach for workout form assessment, leveraging exercise-specific motion patterns to improve error detection in gym videos with limited labeled data.

Contribution

It proposes novel self-supervised methods using pose contrastive learning and motion disentangling tailored for exercise videos, and curates a new annotated dataset for training and evaluation.

Findings

Self-supervised representations outperform standard pose estimators.

The methods generalize to pose estimation and dive quality assessment.

Curated dataset enhances supervised error detection.

Abstract

Maintaining proper form while exercising is important for preventing injuries and maximizing muscle mass gains. Detecting errors in workout form naturally requires estimating human's body pose. However, off-the-shelf pose estimators struggle to perform well on the videos recorded in gym scenarios due to factors such as camera angles, occlusion from gym equipment, illumination, and clothing. To aggravate the problem, the errors to be detected in the workouts are very subtle. To that end, we propose to learn exercise-oriented image and video representations from unlabeled samples such that a small dataset annotated by experts suffices for supervised error detection. In particular, our domain knowledge-informed self-supervised approaches (pose contrastive learning and motion disentangling) exploit the harmonic motion of the exercise actions, and capitalize on the large variances in camera angles, clothes, and illumination to learn powerful representations. To facilitate our self-supervised pretraining, and supervised finetuning, we curated a new exercise dataset, \emph{Fitness-AQA} (\url{https://github.com/ParitoshParmar/Fitness-AQA}), comprising of three exercises: BackSquat, BarbellRow, and OverheadPress. It has been annotated by expert trainers for multiple crucial and typically occurring exercise errors. Experimental results show that our self-supervised representations outperform off-the-shelf 2D- and 3D-pose estimators and several other baselines. We also show that our approaches can be applied to other domains/tasks such as pose estimation and dive quality assessment.