Towards Activated Muscle Group Estimation in the Wild

TL;DR

This paper introduces the MuscleMap dataset and a new model, TransM3E, for estimating active muscle groups during diverse physical activities in real-world videos, addressing generalization challenges.

Contribution

The paper presents the first large-scale dataset for video-based muscle activity estimation and proposes a novel multi-modality model with cross-modal knowledge distillation for improved generalization.

Findings

TransM3E outperforms existing models on seen activities.

The dataset enables new applications in sports and rehabilitation.

Generalization to unseen activities remains challenging.

Abstract

In this paper, we tackle the new task of video-based Activated Muscle Group Estimation (AMGE) aiming at identifying active muscle regions during physical activity in the wild. To this intent, we provide the MuscleMap dataset featuring >15K video clips with 135 different activities and 20 labeled muscle groups. This dataset opens the vistas to multiple video-based applications in sports and rehabilitation medicine under flexible environment constraints. The proposed MuscleMap dataset is constructed with YouTube videos, specifically targeting High-Intensity Interval Training (HIIT) physical exercise in the wild. To make the AMGE model applicable in real-life situations, it is crucial to ensure that the model can generalize well to numerous types of physical activities not present during training and involving new combinations of activated muscles. To achieve this, our benchmark also covers an evaluation setting where the model is exposed to activity types excluded from the training set. Our experiments reveal that the generalizability of existing architectures adapted for the AMGE task remains a challenge. Therefore, we also propose a new approach, TransM3E, which employs a multi-modality feature fusion mechanism between both the video transformer model and the skeleton-based graph convolution model with novel cross-modal knowledge distillation executed on multi-classification tokens. The proposed method surpasses all popular video classification models when dealing with both, previously seen and new types of physical activities. The database and code can be found at https://github.com/KPeng9510/MuscleMap.