CROMOSim: A Deep Learning-based Cross-modality Inertial Measurement Simulator

TL;DR

CROMOSim is a deep learning-based simulator that generates high-fidelity virtual IMU data from motion capture or RGB cameras, aiding in human activity recognition by addressing data scarcity.

Contribution

It introduces a novel cross-modality simulation framework using a DNN to convert imperfect pose estimations into realistic IMU data, enhancing data augmentation for HAR.

Findings

Achieves 6.7% improvement in HAR accuracy over baseline methods.

Demonstrates high fidelity of simulated IMU data for training deep models.

Effectively mitigates data scarcity in wearable sensor applications.

Abstract

With the prevalence of wearable devices, inertial measurement unit (IMU) data has been utilized in monitoring and assessment of human mobility such as human activity recognition (HAR). Training deep neural network (DNN) models for these tasks require a large amount of labeled data, which are hard to acquire in uncontrolled environments. To mitigate the data scarcity problem, we design CROMOSim, a cross-modality sensor simulator that simulates high fidelity virtual IMU sensor data from motion capture systems or monocular RGB cameras. It utilizes a skinned multi-person linear model (SMPL) for 3D body pose and shape representations, to enable simulation from arbitrary on-body positions. A DNN model is trained to learn the functional mapping from imperfect trajectory estimations in a 3D SMPL body tri-mesh due to measurement noise, calibration errors, occlusion and other modeling artifacts, to IMU data. We evaluate the fidelity of CROMOSim simulated data and its utility in data augmentation on various HAR datasets. Extensive experiment results show that the proposed model achieves a 6.7% improvement over baseline methods in a HAR task.