Enhancing Action Recognition by Leveraging the Hierarchical Structure of Actions and Textual Context

TL;DR

This paper introduces a transformer-based approach that leverages hierarchical action structures and textual context to significantly improve action recognition accuracy in complex, real-world datasets.

Contribution

It presents a novel hierarchical action recognition model that combines visual and textual features with a joint loss function, and extends datasets with hierarchical annotations.

Findings

Achieves over 17% improvement in top-1 accuracy on multiple datasets.

Demonstrates the effectiveness of hierarchical and contextual information integration.

Outperforms state-of-the-art methods consistently.

Abstract

We propose a novel approach to improve action recognition by exploiting the hierarchical organization of actions and by incorporating contextualized textual information, including location and previous actions, to reflect the action's temporal context. To achieve this, we introduce a transformer architecture tailored for action recognition that employs both visual and textual features. Visual features are obtained from RGB and optical flow data, while text embeddings represent contextual information. Furthermore, we define a joint loss function to simultaneously train the model for both coarse- and fine-grained action recognition, effectively exploiting the hierarchical nature of actions. To demonstrate the effectiveness of our method, we extend the Toyota Smarthome Untrimmed (TSU) dataset by incorporating action hierarchies, resulting in the Hierarchical TSU dataset, a hierarchical dataset designed for monitoring activities of the elderly in home environments. An ablation study assesses the performance impact of different strategies for integrating contextual and hierarchical data. Experimental results demonstrate that the proposed method consistently outperforms SOTA methods on the Hierarchical TSU dataset, Assembly101 and IkeaASM, achieving over a 17% improvement in top-1 accuracy.