Concurrent Activity Recognition with Multimodal CNN-LSTM Structure

TL;DR

This paper presents a scalable multimodal CNN-LSTM system for recognizing concurrent activities from multisensor data, achieving comparable performance across diverse datasets with a unified model.

Contribution

It introduces the first single-model approach for concurrent activity recognition using multisensory data, combining CNN and LSTM for spatial and temporal feature extraction.

Findings

Achieved performance comparable to domain-specific systems

Successfully recognized concurrent activities in multiple datasets

Demonstrated scalability and simplicity of the model

Abstract

We introduce a system that recognizes concurrent activities from real-world data captured by multiple sensors of different types. The recognition is achieved in two steps. First, we extract spatial and temporal features from the multimodal data. We feed each datatype into a convolutional neural network that extracts spatial features, followed by a long-short term memory network that extracts temporal information in the sensory data. The extracted features are then fused for decision making in the second step. Second, we achieve concurrent activity recognition with a single classifier that encodes a binary output vector in which elements indicate whether the corresponding activity types are currently in progress. We tested our system with three datasets from different domains recorded using different sensors and achieved performance comparable to existing systems designed specifically for those domains. Our system is the first to address the concurrent activity recognition with multisensory data using a single model, which is scalable, simple to train and easy to deploy.