Improving Out-of-distribution Human Activity Recognition via IMU-Video Cross-modal Representation Learning

TL;DR

This paper introduces a cross-modal self-supervised pretraining method using IMU-video data to improve out-of-distribution human activity recognition, especially in clinical settings like Parkinson's disease, demonstrating superior generalization over existing methods.

Contribution

The study presents a novel cross-modal pretraining approach that enhances the generalizability of HAR models to unseen data and environments, outperforming existing IMU-only and IMU-video pretraining techniques.

Findings

Outperforms state-of-the-art IMU-video pretraining in zero-shot and few-shot settings.

Improves generalization of HAR models to out-of-distribution datasets.

Effective in clinical scenarios like Parkinson's disease monitoring.

Abstract

Human Activity Recognition (HAR) based on wearable inertial sensors plays a critical role in remote health monitoring. In patients with movement disorders, the ability to detect abnormal patient movements in their home environments can enable continuous optimization of treatments and help alert caretakers as needed. Machine learning approaches have been proposed for HAR tasks using Inertial Measurement Unit (IMU) data; however, most rely on application-specific labels and lack generalizability to data collected in different environments or populations. To address this limitation, we propose a new cross-modal self-supervised pretraining approach to learn representations from large-sale unlabeled IMU-video data and demonstrate improved generalizability in HAR tasks on out of distribution (OOD) IMU datasets, including a dataset collected from patients with Parkinson's disease. Specifically, our results indicate that the proposed cross-modal pretraining approach outperforms the current state-of-the-art IMU-video pretraining approach and IMU-only pretraining under zero-shot and few-shot evaluations. Broadly, our study provides evidence that in highly dynamic data modalities, such as IMU signals, cross-modal pretraining may be a useful tool to learn generalizable data representations. Our software is available at https://github.com/scheshmi/IMU-Video-OOD-HAR.