Multidomain Multimodal Fusion For Human Action Recognition Using Inertial Sensors

TL;DR

This paper introduces a novel multidomain multimodal fusion framework that enhances human action recognition accuracy by extracting and combining complementary features from inertial sensor data transformed into multiple domains.

Contribution

It proposes a new fusion method that transforms inertial data into spatial, frequency, and time-spectrum domains and fuses features using Canonical Correlation based Fusion, improving recognition performance.

Findings

Outperforms state-of-the-art methods on three inertial datasets.

Effectively extracts complementary features across multiple domains.

Demonstrates significant accuracy improvements in action recognition.

Abstract

One of the major reasons for misclassification of multiplex actions during action recognition is the unavailability of complementary features that provide the semantic information about the actions. In different domains these features are present with different scales and intensities. In existing literature, features are extracted independently in different domains, but the benefits from fusing these multidomain features are not realized. To address this challenge and to extract complete set of complementary information, in this paper, we propose a novel multidomain multimodal fusion framework that extracts complementary and distinct features from different domains of the input modality. We transform input inertial data into signal images, and then make the input modality multidomain and multimodal by transforming spatial domain information into frequency and time-spectrum domain using Discrete Fourier Transform (DFT) and Gabor wavelet transform (GWT) respectively. Features in different domains are extracted by Convolutional Neural networks (CNNs) and then fused by Canonical Correlation based Fusion (CCF) for improving the accuracy of human action recognition. Experimental results on three inertial datasets show the superiority of the proposed method when compared to the state-of-the-art.