An Examination of Wearable Sensors and Video Data Capture for Human Exercise Classification

TL;DR

This study compares wearable sensor and video-based methods for classifying human exercises, finding that video can outperform a single IMU and that combining both modalities yields the best results, offering a practical approach using common devices.

Contribution

The paper demonstrates that a single camera can outperform a single IMU in exercise classification and that combining video with minimal sensors enhances accuracy, reducing practical limitations.

Findings

Single camera outperforms a single IMU by 10 percentage points.

At least 3 IMUs are needed to outperform a single camera.

Multivariate time series classifiers on raw data outperform traditional feature-based methods.

Abstract

Wearable sensors such as Inertial Measurement Units (IMUs) are often used to assess the performance of human exercise. Common approaches use handcrafted features based on domain expertise or automatically extracted features using time series analysis. Multiple sensors are required to achieve high classification accuracy, which is not very practical. These sensors require calibration and synchronization and may lead to discomfort over longer time periods. Recent work utilizing computer vision techniques has shown similar performance using video, without the need for manual feature engineering, and avoiding some pitfalls such as sensor calibration and placement on the body. In this paper, we compare the performance of IMUs to a video-based approach for human exercise classification on two real-world datasets consisting of Military Press and Rowing exercises. We compare the performance using a single camera that captures video in the frontal view versus using 5 IMUs placed on different parts of the body. We observe that an approach based on a single camera can outperform a single IMU by 10 percentage points on average. Additionally, a minimum of 3 IMUs are required to outperform a single camera. We observe that working with the raw data using multivariate time series classifiers outperforms traditional approaches based on handcrafted or automatically extracted features. Finally, we show that an ensemble model combining the data from a single camera with a single IMU outperforms either data modality. Our work opens up new and more realistic avenues for this application, where a video captured using a readily available smartphone camera, combined with a single sensor, can be used for effective human exercise classification.