Part-level Action Parsing via a Pose-guided Coarse-to-Fine Framework

TL;DR

This paper introduces a pose-guided, coarse-to-fine framework for Part-level Action Parsing that predicts both video-level actions and frame-level body part actions, achieving state-of-the-art results on Kinetics-TPS.

Contribution

It proposes a novel pose-guided positional embedding and segment-level feature recognition for accurate, explainable part-level action parsing in videos.

Findings

Achieves 31.10% ROC score on Kinetics-TPS

Outperforms existing methods significantly

Balances accuracy and computation effectively

Abstract

Action recognition from videos, i.e., classifying a video into one of the pre-defined action types, has been a popular topic in the communities of artificial intelligence, multimedia, and signal processing. However, existing methods usually consider an input video as a whole and learn models, e.g., Convolutional Neural Networks (CNNs), with coarse video-level class labels. These methods can only output an action class for the video, but cannot provide fine-grained and explainable cues to answer why the video shows a specific action. Therefore, researchers start to focus on a new task, Part-level Action Parsing (PAP), which aims to not only predict the video-level action but also recognize the frame-level fine-grained actions or interactions of body parts for each person in the video. To this end, we propose a coarse-to-fine framework for this challenging task. In particular, our framework first predicts the video-level class of the input video, then localizes the body parts and predicts the part-level action. Moreover, to balance the accuracy and computation in part-level action parsing, we propose to recognize the part-level actions by segment-level features. Furthermore, to overcome the ambiguity of body parts, we propose a pose-guided positional embedding method to accurately localize body parts. Through comprehensive experiments on a large-scale dataset, i.e., Kinetics-TPS, our framework achieves state-of-the-art performance and outperforms existing methods over a 31.10% ROC score.