Inertial Hallucinations -- When Wearable Inertial Devices Start Seeing Things

TL;DR

This paper introduces a multimodal sensor fusion framework using privileged information to improve inertial sensor-based activity recognition, achieving state-of-the-art accuracy on benchmark datasets.

Contribution

The paper presents a novel fusion approach combining modality hallucination and triplet learning to handle missing sensors and improve inertial data classification.

Findings

Achieved 6.6% accuracy improvement on UTD-MHAD dataset

Achieved 5.5% accuracy improvement on Berkeley MHAD dataset

Established new state-of-the-art inertial-only classification results

Abstract

We propose a novel approach to multimodal sensor fusion for Ambient Assisted Living (AAL) which takes advantage of learning using privileged information (LUPI). We address two major shortcomings of standard multimodal approaches, limited area coverage and reduced reliability. Our new framework fuses the concept of modality hallucination with triplet learning to train a model with different modalities to handle missing sensors at inference time. We evaluate the proposed model on inertial data from a wearable accelerometer device, using RGB videos and skeletons as privileged modalities, and show an improvement of accuracy of an average 6.6% on the UTD-MHAD dataset and an average 5.5% on the Berkeley MHAD dataset, reaching a new state-of-the-art for inertial-only classification accuracy on these datasets. We validate our framework through several ablation studies.