Decomposing and Fusing Intra- and Inter-Sensor Spatio-Temporal Signal for Multi-Sensor Wearable Human Activity Recognition

TL;DR

This paper introduces DecomposeWHAR, a novel multi-sensor human activity recognition model that effectively captures intra- and inter-sensor spatio-temporal relationships through decomposition and dynamic fusion, outperforming existing methods.

Contribution

The paper proposes a new model that decomposes sensor signals and fuses features using advanced techniques like Depth Separable Convolution, SSM, and self-attention for improved WHAR accuracy.

Findings

Significantly outperforms state-of-the-art models on three datasets.

Effectively captures local and long-range temporal features.

Maintains computational efficiency while improving accuracy.

Abstract

Wearable Human Activity Recognition (WHAR) is a prominent research area within ubiquitous computing. Multi-sensor synchronous measurement has proven to be more effective for WHAR than using a single sensor. However, existing WHAR methods use shared convolutional kernels for indiscriminate temporal feature extraction across each sensor variable, which fails to effectively capture spatio-temporal relationships of intra-sensor and inter-sensor variables. We propose the DecomposeWHAR model consisting of a decomposition phase and a fusion phase to better model the relationships between modality variables. The decomposition creates high-dimensional representations of each intra-sensor variable through the improved Depth Separable Convolution to capture local temporal features while preserving their unique characteristics. The fusion phase begins by capturing relationships between intra-sensor variables and fusing their features at both the channel and variable levels. Long-range temporal dependencies are modeled using the State Space Model (SSM), and later cross-sensor interactions are dynamically captured through a self-attention mechanism, highlighting inter-sensor spatial correlations. Our model demonstrates superior performance on three widely used WHAR datasets, significantly outperforming state-of-the-art models while maintaining acceptable computational efficiency.