A Comparative Study of Human Activity Recognition: Motion, Tactile, and multi-modal Approaches

TL;DR

This paper compares vision-based tactile sensors and IMU-based data gloves for human activity recognition, proposing a multi-modal framework that combines both to improve accuracy in collaborative robotics.

Contribution

It introduces a multi-modal HAR framework integrating tactile and motion data, demonstrating improved performance over single-modality approaches.

Findings

Multi-modal approach outperforms single-modality methods

Tactile sensors effectively classify human activities

Combining tactile and motion data enhances HAR accuracy

Abstract

Human activity recognition (HAR) is essential for effective Human-Robot Collaboration (HRC), enabling robots to interpret and respond to human actions. This study evaluates the ability of a vision-based tactile sensor to classify 15 activities, comparing its performance to an IMU-based data glove. Additionally, we propose a multi-modal framework combining tactile and motion data to leverage their complementary strengths. We examined three approaches: motion-based classification (MBC) using IMU data, tactile-based classification (TBC) with single or dual video streams, and multi-modal classification (MMC) integrating both. Offline validation on segmented datasets assessed each configuration's accuracy under controlled conditions, while online validation on continuous action sequences tested online performance. Results showed the multi-modal approach consistently outperformed single-modality methods, highlighting the potential of integrating tactile and motion sensing to enhance HAR systems for collaborative robotics.