Augmenting Deep Learning Adaptation for Wearable Sensor Data through Combined Temporal-Frequency Image Encoding

TL;DR

This paper introduces a novel combined temporal-frequency image encoding for wearable sensor data, enhancing deep learning-based classification and domain adaptation by integrating frequency domain information with recurrent plots.

Contribution

It proposes a new modified-recurrent plot encoding that fuses temporal and frequency data, along with a Fourier transform-based estimation and mixup augmentation, improving wearable sensor data classification.

Findings

Superior performance on accelerometer activity recognition data

Effective integration of temporal and frequency information

Enhanced domain adaptation capabilities

Abstract

Deep learning advancements have revolutionized scalable classification in many domains including computer vision. However, when it comes to wearable-based classification and domain adaptation, existing computer vision-based deep learning architectures and pretrained models trained on thousands of labeled images for months fall short. This is primarily because wearable sensor data necessitates sensor-specific preprocessing, architectural modification, and extensive data collection. To overcome these challenges, researchers have proposed encoding of wearable temporal sensor data in images using recurrent plots. In this paper, we present a novel modified-recurrent plot-based image representation that seamlessly integrates both temporal and frequency domain information. Our approach incorporates an efficient Fourier transform-based frequency domain angular difference estimation scheme in conjunction with the existing temporal recurrent plot image. Furthermore, we employ mixup image augmentation to enhance the representation. We evaluate the proposed method using accelerometer-based activity recognition data and a pretrained ResNet model, and demonstrate its superior performance compared to existing approaches.