Multilevel neural networks with dual-stage feature fusion for human activity recognition

TL;DR

This paper proposes a novel two-level neural network architecture with dual-stage feature fusion for human activity recognition, improving accuracy by effectively combining multiple network outputs and features.

Contribution

It introduces a new multilevel network framework with late and intermediate fusion, optimizing the integration of CNNs, LSTMs, and convolutional LSTMs for HAR.

Findings

Fusion architectures with both late and intermediate fusion outperform late fusion alone.

The optimal configuration surpasses baseline models in accuracy on benchmark datasets.

Experimental results validate the effectiveness of the proposed dual-stage feature fusion approach.

Abstract

Human activity recognition (HAR) refers to the process of identifying human actions and activities using data collected from sensors. Neural networks, such as convolutional neural networks (CNNs), long short-term memory (LSTM) networks, convolutional LSTM, and their hybrid combinations, have demonstrated exceptional performance in various research domains. Developing a multilevel individual or hybrid model for HAR involves strategically integrating multiple networks to capitalize on their complementary strengths. The structural arrangement of these components is a critical factor influencing the overall performance. This study explores a novel framework of a two-level network architecture with dual-stage feature fusion: late fusion, which combines the outputs from the first network level, and intermediate fusion, which integrates the features from both the first and second levels. We evaluated $15$ different network architectures of CNNs, LSTMs, and convolutional LSTMs, incorporating late fusion with and without intermediate fusion, to identify the optimal configuration. Experimental evaluation on two public benchmark datasets demonstrates that architectures incorporating both late and intermediate fusion achieve higher accuracy than those relying on late fusion alone. Moreover, the optimal configuration outperforms baseline models, thereby validating its effectiveness for HAR.