MESEN: Exploit Multimodal Data to Design Unimodal Human Activity Recognition with Few Labels

TL;DR

MESEN leverages unlabeled multimodal data during training to significantly improve unimodal human activity recognition performance with minimal labeled data, addressing modality and label scarcity challenges.

Contribution

Introduces MESEN, a novel multimodal-empowered unimodal HAR framework utilizing cross-modal contrastive learning and pseudo-classification for effective feature extraction.

Findings

MESEN outperforms state-of-the-art methods on eight datasets.

Significant accuracy improvements with few labeled samples.

Effective exploitation of unlabeled multimodal data enhances unimodal HAR.

Abstract

Human activity recognition (HAR) will be an essential function of various emerging applications. However, HAR typically encounters challenges related to modality limitations and label scarcity, leading to an application gap between current solutions and real-world requirements. In this work, we propose MESEN, a multimodal-empowered unimodal sensing framework, to utilize unlabeled multimodal data available during the HAR model design phase for unimodal HAR enhancement during the deployment phase. From a study on the impact of supervised multimodal fusion on unimodal feature extraction, MESEN is designed to feature a multi-task mechanism during the multimodal-aided pre-training stage. With the proposed mechanism integrating cross-modal feature contrastive learning and multimodal pseudo-classification aligning, MESEN exploits unlabeled multimodal data to extract effective unimodal features for each modality. Subsequently, MESEN can adapt to downstream unimodal HAR with only a few labeled samples. Extensive experiments on eight public multimodal datasets demonstrate that MESEN achieves significant performance improvements over state-of-the-art baselines in enhancing unimodal HAR by exploiting multimodal data.