C3T: Cross-modal Transfer Through Time for Sensor-based Human Activity Recognition

TL;DR

This paper introduces C3T, a novel method for transferring knowledge across different sensor modalities in human activity recognition by preserving temporal information, significantly improving accuracy and robustness in unsupervised modality adaptation.

Contribution

C3T is the first approach to align temporal latent vectors across modalities, enhancing transferability and robustness in sensor-based HAR under unsupervised conditions.

Findings

C3T outperforms existing UMA methods by at least 8% in accuracy.

C3T demonstrates superior robustness to temporal distortions like time-shift and dilation.

C3T shows promise for generalizable time-series sensor models.

Abstract

In order to unlock the potential of diverse sensors, we investigate a method to transfer knowledge between time-series modalities using a multimodal \textit{temporal} representation space for Human Activity Recognition (HAR). Specifically, we explore the setting where the modality used in testing has no labeled data during training, which we refer to as Unsupervised Modality Adaptation (UMA). We categorize existing UMA approaches as Student-Teacher or Contrastive Alignment methods. These methods typically compress continuous-time data samples into single latent vectors during alignment, inhibiting their ability to transfer temporal information through real-world temporal distortions. To address this, we introduce Cross-modal Transfer Through Time (C3T), which preserves temporal information during alignment to handle dynamic sensor data better. C3T achieves this by aligning a set of temporal latent vectors across sensing modalities. Our extensive experiments on various camera+IMU datasets demonstrate that C3T outperforms existing methods in UMA by at least 8% in accuracy and shows superior robustness to temporal distortions such as time-shift, misalignment, and dilation. Our findings suggest that C3T has significant potential for developing generalizable models for time-series sensor data, opening new avenues for various multimodal applications.