FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding

TL;DR

FineGym is a comprehensive hierarchical video dataset designed to improve fine-grained action recognition, especially in sports, by providing detailed temporal annotations at multiple semantic levels, challenging existing methods to parse complex activity structures.

Contribution

The paper introduces FineGym, a novel gymnastic video dataset with hierarchical annotations at action and sub-action levels, enabling more detailed and challenging action recognition research.

Findings

Existing methods struggle with fine-grained temporal parsing.

Hierarchical annotations reveal challenges in subtle action differentiation.

Systematic evaluation provides insights into current method limitations.

Abstract

On public benchmarks, current action recognition techniques have achieved great success. However, when used in real-world applications, e.g. sport analysis, which requires the capability of parsing an activity into phases and differentiating between subtly different actions, their performances remain far from being satisfactory. To take action recognition to a new level, we develop FineGym, a new dataset built on top of gymnastic videos. Compared to existing action recognition datasets, FineGym is distinguished in richness, quality, and diversity. In particular, it provides temporal annotations at both action and sub-action levels with a three-level semantic hierarchy. For example, a "balance beam" event will be annotated as a sequence of elementary sub-actions derived from five sets: "leap-jump-hop", "beam-turns", "flight-salto", "flight-handspring", and "dismount", where the sub-action in each set will be further annotated with finely defined class labels. This new level of granularity presents significant challenges for action recognition, e.g. how to parse the temporal structures from a coherent action, and how to distinguish between subtly different action classes. We systematically investigate representative methods on this dataset and obtain a number of interesting findings. We hope this dataset could advance research towards action understanding.