Action100M: A Large-scale Video Action Dataset

TL;DR

Action100M is a comprehensive large-scale video dataset with detailed annotations, designed to advance open-vocabulary action recognition and understanding in machine learning models.

Contribution

The paper introduces Action100M, a fully automated pipeline for creating a large-scale, richly annotated video dataset from internet videos, enabling scalable research in video understanding.

Findings

Training VL-JEPA on Action100M improves zero-shot action recognition performance.

Action100M provides extensive open-vocabulary annotations for diverse actions.

The dataset supports scalable and detailed video understanding research.

Abstract

Inferring physical actions from visual observations is a fundamental capability for advancing machine intelligence in the physical world. Achieving this requires large-scale, open-vocabulary video action datasets that span broad domains. We introduce Action100M, a large-scale dataset constructed from 1.2M Internet instructional videos (14.6 years of duration), yielding O(100 million) temporally localized segments with open-vocabulary action supervision and rich captions. Action100M is generated by a fully automated pipeline that (i) performs hierarchical temporal segmentation using V-JEPA 2 embeddings, (ii) produces multi-level frame and segment captions organized as a Tree-of-Captions, and (iii) aggregates evidence with a reasoning model (GPT-OSS-120B) under a multi-round Self-Refine procedure to output structured annotations (brief/detailed action, actor, brief/detailed caption). Training VL-JEPA on Action100M demonstrates consistent data-scaling improvements and strong zero-shot performance across diverse action recognition benchmarks, establishing Action100M as a new foundation for scalable research in video understanding and world modeling.