Video2IMU: Realistic IMU features and signals from videos

TL;DR

This paper introduces a neural network-based method to generate realistic wearable sensor signals from videos, enabling training of HAR models without extensive real sensor data, demonstrated on an industrial safety dataset.

Contribution

It presents a novel approach to synthesize wearable sensor signals from videos, reducing the need for large labeled sensor datasets in HAR.

Findings

Generated signals are realistic and comparable to real sensor data.

HAR models trained on synthetic signals perform similarly to those trained on real data.

Method enables leveraging labeled video data for sensor-based activity recognition.

Abstract

Human Activity Recognition (HAR) from wearable sensor data identifies movements or activities in unconstrained environments. HAR is a challenging problem as it presents great variability across subjects. Obtaining large amounts of labelled data is not straightforward, since wearable sensor signals are not easy to label upon simple human inspection. In our work, we propose the use of neural networks for the generation of realistic signals and features using human activity monocular videos. We show how these generated features and signals can be utilized, instead of their real counterparts, to train HAR models that can recognize activities using signals obtained with wearable sensors. To prove the validity of our methods, we perform experiments on an activity recognition dataset created for the improvement of industrial work safety. We show that our model is able to realistically generate virtual sensor signals and features usable to train a HAR classifier with comparable performance as the one trained using real sensor data. Our results enable the use of available, labelled video data for training HAR models to classify signals from wearable sensors.