HieroAction: Hierarchically Guided VLM for Fine-Grained Action Analysis

TL;DR

HieroAction is a hierarchical vision-language model that provides interpretable, step-by-step assessments of human actions, combining structured reasoning with reinforcement learning to improve accuracy in action evaluation tasks.

Contribution

The paper introduces HieroAction, a novel model that integrates stepwise action reasoning and hierarchical policy learning for fine-grained, interpretable action assessment.

Findings

Outperforms existing methods on multiple benchmark datasets.

Provides structured, interpretable evaluations of human actions.

Enhances scoring accuracy through reinforcement learning.

Abstract

Evaluating human actions with clear and detailed feedback is important in areas such as sports, healthcare, and robotics, where decisions rely not only on final outcomes but also on interpretable reasoning. However, most existing methods provide only a final score without explanation or detailed analysis, limiting their practical applicability. To address this, we introduce HieroAction, a vision-language model that delivers accurate and structured assessments of human actions. HieroAction builds on two key ideas: (1) Stepwise Action Reasoning, a tailored chain of thought process designed specifically for action assessment, which guides the model to evaluate actions step by step, from overall recognition through sub action analysis to final scoring, thus enhancing interpretability and structured understanding; and (2) Hierarchical Policy Learning, a reinforcement learning strategy that enables the model to learn fine grained sub action dynamics and align them with high level action quality, thereby improving scoring precision. The reasoning pathway structures the evaluation process, while policy learning refines each stage through reward based optimization. Their integration ensures accurate and interpretable assessments, as demonstrated by superior performance across multiple benchmark datasets. Code will be released upon acceptance.