EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understanding

TL;DR

EgoExo-Fitness introduces a comprehensive dataset with synchronized egocentric and exocentric videos, rich annotations, and benchmarks for full-body action understanding, enabling advances in multi-view action analysis and interpretation.

Contribution

The paper presents a novel dataset with synchronized multi-view videos, detailed annotations, and benchmarks for egocentric and exocentric full-body action understanding, including new interpretability and quality assessment tasks.

Findings

New dataset with rich annotations and synchronized views

Benchmarks for multiple action understanding tasks

Analysis demonstrating dataset utility and potential applications

Abstract

We present EgoExo-Fitness, a new full-body action understanding dataset, featuring fitness sequence videos recorded from synchronized egocentric and fixed exocentric (third-person) cameras. Compared with existing full-body action understanding datasets, EgoExo-Fitness not only contains videos from first-person perspectives, but also provides rich annotations. Specifically, two-level temporal boundaries are provided to localize single action videos along with sub-steps of each action. More importantly, EgoExo-Fitness introduces innovative annotations for interpretable action judgement--including technical keypoint verification, natural language comments on action execution, and action quality scores. Combining all of these, EgoExo-Fitness provides new resources to study egocentric and exocentric full-body action understanding across dimensions of "what", "when", and "how well". To facilitate research on egocentric and exocentric full-body action understanding, we construct benchmarks on a suite of tasks (i.e., action classification, action localization, cross-view sequence verification, cross-view skill determination, and a newly proposed task of guidance-based execution verification), together with detailed analysis. Code and data will be available at https://github.com/iSEE-Laboratory/EgoExo-Fitness/tree/main.