Human Stone Toolmaking Action Grammar (HSTAG): A Challenging Benchmark for Fine-grained Motor Behavior Recognition

TL;DR

HSTAG is a new, richly annotated video dataset of expert stone toolmaking, designed to challenge and advance fine-grained motor behavior recognition algorithms with its complex, rapid, and variable actions.

Contribution

The paper introduces HSTAG, a novel dataset with detailed annotations of stone toolmaking behaviors, addressing the lack of domain-specific benchmarks for complex motor actions.

Findings

Existing models face challenges with HSTAG's rapid, variable actions.

HSTAG's complexity reveals limitations of current action recognition algorithms.

The dataset highlights the need for more sophisticated models for fine-grained motor behavior recognition.

Abstract

Action recognition has witnessed the development of a growing number of novel algorithms and datasets in the past decade. However, the majority of public benchmarks were constructed around activities of daily living and annotated at a rather coarse-grained level, which lacks diversity in domain-specific datasets, especially for rarely seen domains. In this paper, we introduced Human Stone Toolmaking Action Grammar (HSTAG), a meticulously annotated video dataset showcasing previously undocumented stone toolmaking behaviors, which can be used for investigating the applications of advanced artificial intelligence techniques in understanding a rapid succession of complex interactions between two hand-held objects. HSTAG consists of 18,739 video clips that record 4.5 hours of experts' activities in stone toolmaking. Its unique features include (i) brief action durations and frequent transitions, mirroring the rapid changes inherent in many motor behaviors; (ii) multiple angles of view and switches among multiple tools, increasing intra-class variability; (iii) unbalanced class distributions and high similarity among different action sequences, adding difficulty in capturing distinct patterns for each action. Several mainstream action recognition models are used to conduct experimental analysis, which showcases the challenges and uniqueness of HSTAG https://nyu.databrary.org/volume/1697.