DenseStep2M: A Scalable, Training-Free Pipeline for Dense Instructional Video Annotation

TL;DR

DenseStep2M is a large-scale, training-free pipeline that automatically extracts high-quality procedural annotations from instructional videos, enabling improved long-term video understanding and downstream task performance.

Contribution

We introduce DenseStep2M, a novel automated pipeline that generates a comprehensive instructional video dataset without training, leveraging multimodal models for high-quality annotation and evaluation.

Findings

Models trained on DenseStep2M show improved captioning and localization performance.

The dataset enables robust zero-shot generalization across different video perspectives.

Auto-generated steps align well with human annotations in evaluation.

Abstract

Long-term video understanding requires interpreting complex temporal events and reasoning over procedural activities. While instructional video corpora, like HowTo100M, offer rich resources for model training, they present significant challenges, including noisy ASR transcripts and inconsistent temporal alignments between narration and visual content. In this work, we introduce an automated, training-free pipeline to extract high-quality procedural annotations from in-the-wild instructional videos. Our approach segments videos into coherent shots, filters poorly aligned content, and leverages state-of-the-art multimodal and large language models (Qwen2.5-VL and DeepSeek-R1) to generate structured, temporally grounded procedural steps. This pipeline yields DenseStep2M, a large-scale dataset comprising approximately 100K videos and 2M detailed instructional steps, designed to support comprehensive long-form video understanding. To rigorously evaluate our pipeline, we curate DenseCaption100, a benchmark of high-quality, human-written captions. Evaluations demonstrate strong alignment between our auto-generated steps and human annotations. Furthermore, we validate the utility of DenseStep2M across three core downstream tasks: dense video captioning, procedural step grounding, and cross-modal retrieval. Models fine-tuned on DenseStep2M achieve substantial gains in captioning quality and temporal localization, while exhibiting robust zero-shot generalization across egocentric, exocentric, and mixed-perspective domains. These results underscore the effectiveness of DenseStep2M in facilitating advanced multimodal alignment and long-term activity reasoning. Our dataset is available at https://huggingface.co/datasets/mingjige/DenseStep2M.