View-invariant Deep Architecture for Human Action Recognition using late fusion

TL;DR

This paper introduces a view-invariant deep learning framework for human action recognition that combines motion and shape dynamics, utilizing late fusion of multiple streams, and demonstrates superior performance on multiple benchmarks.

Contribution

A novel view-invariant deep architecture integrating motion and shape dynamics with late fusion for improved action recognition accuracy.

Findings

Outperforms existing state-of-the-art methods significantly.

Effective in cross-view and cross-subject validation schemes.

Achieves higher accuracy, ROC, and AUC metrics.

Abstract

Human action Recognition for unknown views is a challenging task. We propose a view-invariant deep human action recognition framework, which is a novel integration of two important action cues: motion and shape temporal dynamics (STD). The motion stream encapsulates the motion content of action as RGB Dynamic Images (RGB-DIs) which are processed by the fine-tuned InceptionV3 model. The STD stream learns long-term view-invariant shape dynamics of action using human pose model (HPM) based view-invariant features mined from structural similarity index matrix (SSIM) based key depth human pose frames. To predict the score of the test sample, three types of late fusion (maximum, average and product) techniques are applied on individual stream scores. To validate the performance of the proposed novel framework the experiments are performed using both cross subject and cross-view validation schemes on three publically available benchmarks- NUCLA multi-view dataset, UWA3D-II Activity dataset and NTU RGB-D Activity dataset. Our algorithm outperforms with existing state-of-the-arts significantly that is reported in terms of accuracy, receiver operating characteristic (ROC) curve and area under the curve (AUC).