DynaStride: Dynamic Stride Windowing with MMCoT for Instructional Multi-Scene Captioning

TL;DR

DynaStride is a novel pipeline that generates coherent scene-level captions for instructional videos by adaptively sampling frames and employing multimodal reasoning, improving temporal coherence and informativeness without manual scene segmentation.

Contribution

The paper introduces DynaStride, a new method that automatically produces scene-level captions in instructional videos using adaptive sampling and multimodal reasoning, eliminating the need for manual scene segmentation.

Findings

DynaStride outperforms strong baselines on BLEU, METEOR, BERTScore, and CLIPScore.

Captions generated by DynaStride are more temporally coherent and informative.

Empirical results demonstrate consistent improvements across multiple evaluation metrics.

Abstract

Scene-level captioning in instructional videos can enhance learning by requiring an understanding of both visual cues and temporal structure. By aligning visual cues with textual guidance, this understanding supports procedural learning and multimodal reasoning, providing a richer context for skill acquisition. However, captions that fail to capture this structure may lack coherence and quality, which can create confusion and undermine the video's educational intent. To address this gap, we introduce DynaStride, a pipeline to generate coherent, scene-level captions without requiring manual scene segmentation. Using the YouCookII dataset's scene annotations, DynaStride performs adaptive frame sampling and multimodal windowing to capture key transitions within each scene. It then employs a multimodal chain-of-thought process to produce multiple action-object pairs, which are refined and fused using a dynamic stride window selection algorithm that adaptively balances temporal context and redundancy. The final scene-level caption integrates visual semantics and temporal reasoning in a single instructional caption. Empirical evaluations against strong baselines, including VLLaMA3 and GPT-4o, demonstrate consistent gains on both N-gram-based metrics (BLEU, METEOR) and semantic similarity measures (BERTScore, CLIPScore). Qualitative analyses further show that DynaStride produces captions that are more temporally coherent and informative, suggesting a promising direction for improving AI-powered instructional content generation.