Fine-grained Human Activity Recognition Using Virtual On-body Acceleration Data

TL;DR

This paper evaluates the effectiveness of virtual accelerometry data from videos for fine-grained human activity recognition, introducing a measure to assess movement subtlety and testing the limits of the IMUTube transfer approach.

Contribution

It introduces the Motion Subtlety Index (MSI) to quantify activity difficulty and performs a stress-test on IMUTube to determine its applicability to subtle movements.

Findings

MSI correlates with activity recognition accuracy.

IMUTube's effectiveness decreases with subtle movements.

The study maps the limits of cross-modality transfer for HAR.

Abstract

Previous work has demonstrated that virtual accelerometry data, extracted from videos using cross-modality transfer approaches like IMUTube, is beneficial for training complex and effective human activity recognition (HAR) models. Systems like IMUTube were originally designed to cover activities that are based on substantial body (part) movements. Yet, life is complex, and a range of activities of daily living is based on only rather subtle movements, which bears the question to what extent systems like IMUTube are of value also for fine-grained HAR, i.e., When does IMUTube break? In this work we first introduce a measure to quantitatively assess the subtlety of human movements that are underlying activities of interest--the motion subtlety index (MSI)--which captures local pixel movements and pose changes in the vicinity of target virtual sensor locations, and correlate it to the eventual activity recognition accuracy. We then perform a "stress-test" on IMUTube and explore for which activities with underlying subtle movements a cross-modality transfer approach works, and for which not. As such, the work presented in this paper allows us to map out the landscape for IMUTube applications in practical scenarios.