Tri-axial Self-Attention for Concurrent Activity Recognition

TL;DR

This paper introduces a novel transformer-based system for concurrent activity recognition that uses feature-to-activity attention and association masks to improve detection accuracy and interpretability.

Contribution

It proposes a new self-attention mechanism with an association mask and a feature-to-activity attention for better concurrent activity recognition.

Findings

Achieved state-of-the-art performance on three datasets.

Demonstrated ability to locate important spatial-temporal features.

Applicable to general multilabel classification problems.

Abstract

We present a system for concurrent activity recognition. To extract features associated with different activities, we propose a feature-to-activity attention that maps the extracted global features to sub-features associated with individual activities. To model the temporal associations of individual activities, we propose a transformer-network encoder that models independent temporal associations for each activity. To make the concurrent activity prediction aware of the potential associations between activities, we propose self-attention with an association mask. Our system achieved state-of-the-art or comparable performance on three commonly used concurrent activity detection datasets. Our visualizations demonstrate that our system is able to locate the important spatial-temporal features for final decision making. We also showed that our system can be applied to general multilabel classification problems.