ActionPrompt: Action-Guided 3D Human Pose Estimation With Text and Pose Prompting

TL;DR

This paper introduces Action Prompt Module (APM), a versatile plug-and-play approach that leverages action-related textual and pose prompts to enhance 3D human pose estimation from videos, significantly improving existing frameworks.

Contribution

The paper proposes a novel, adaptable module that incorporates action labels and pose patterns to improve 3D human pose estimation accuracy across different models.

Findings

APM improves 3D HPE performance across multiple frameworks.

Action-related text prompts effectively transfer language knowledge to pose estimation.

Pose prompts help refine pose predictions by exploiting action-specific patterns.

Abstract

Recent 2D-to-3D human pose estimation (HPE) utilizes temporal consistency across sequences to alleviate the depth ambiguity problem but ignore the action related prior knowledge hidden in the pose sequence. In this paper, we propose a plug-and-play module named Action Prompt Module (APM) that effectively mines different kinds of action clues for 3D HPE. The highlight is that, the mining scheme of APM can be widely adapted to different frameworks and bring consistent benefits. Specifically, we first present a novel Action-related Text Prompt module (ATP) that directly embeds action labels and transfers the rich language information in the label to the pose sequence. Besides, we further introduce Action-specific Pose Prompt module (APP) to mine the position-aware pose pattern of each action, and exploit the correlation between the mined patterns and input pose sequence for further pose refinement. Experiments show that APM can improve the performance of most video-based 2D-to-3D HPE frameworks by a large margin.