PresSim: An End-to-end Framework for Dynamic Ground Pressure Profile Generation from Monocular Videos Using Physics-based 3D Simulation

TL;DR

PresSim is an innovative end-to-end framework that synthesizes human ground pressure profiles from monocular videos using physics-based simulation, significantly reducing the need for extensive sensor data collection in human activity recognition.

Contribution

It introduces a novel multi-stage process combining computer vision, physics simulation, and deep learning to generate pressure data from videos, advancing unobtrusive activity sensing.

Findings

Successfully validated with yoga sequences and pressure mats.

Achieved realistic pressure profile synthesis from monocular videos.

Reduced data collection effort for human activity recognition.

Abstract

Ground pressure exerted by the human body is a valuable source of information for human activity recognition (HAR) in unobtrusive pervasive sensing. While data collection from pressure sensors to develop HAR solutions requires significant resources and effort, we present a novel end-to-end framework, PresSim, to synthesize sensor data from videos of human activities to reduce such effort significantly. PresSim adopts a 3-stage process: first, extract the 3D activity information from videos with computer vision architectures; then simulate the floor mesh deformation profiles based on the 3D activity information and gravity-included physics simulation; lastly, generate the simulated pressure sensor data with deep learning models. We explored two approaches for the 3D activity information: inverse kinematics with mesh re-targeting, and volumetric pose and shape estimation. We validated PresSim with an experimental setup with a monocular camera to provide input and a pressure-sensing fitness mat (80x28 spatial resolution) to provide the sensor ground truth, where nine participants performed a set of predefined yoga sequences.