SEZ-HARN: Self-Explainable Zero-shot Human Activity Recognition Network

TL;DR

SEZ-HARN is a novel IMU-based zero-shot human activity recognition model that not only accurately recognizes unseen activities but also provides visual explanations of its decisions, enhancing transparency in HAR applications.

Contribution

It introduces the first self-explainable zero-shot HAR model capable of recognizing unseen activities and explaining its decisions through skeleton videos.

Findings

Achieves near state-of-the-art zero-shot recognition accuracy

Provides realistic and understandable skeleton video explanations

Maintains competitive performance across multiple benchmark datasets

Abstract

Human Activity Recognition (HAR), which uses data from Inertial Measurement Unit (IMU) sensors, has many practical applications in healthcare and assisted living environments. However, its use in real-world scenarios has been limited by the lack of comprehensive IMU-based HAR datasets that cover a wide range of activities and the lack of transparency in existing HAR models. Zero-shot HAR (ZS-HAR) overcomes the data limitations, but current models struggle to explain their decisions, making them less transparent. This paper introduces a novel IMU-based ZS-HAR model called the Self-Explainable Zero-shot Human Activity Recognition Network (SEZ-HARN). It can recognize activities not encountered during training and provide skeleton videos to explain its decision-making process. We evaluate the effectiveness of the proposed SEZ-HARN on four benchmark datasets PAMAP2, DaLiAc, HTD-MHAD and MHealth and compare its performance against three state-of-the-art black-box ZS-HAR models. The experiment results demonstrate that SEZ-HARN produces realistic and understandable explanations while achieving competitive Zero-shot recognition accuracy. SEZ-HARN achieves a Zero-shot prediction accuracy within 3\% of the best-performing black-box model on PAMAP2 while maintaining comparable performance on the other three datasets.