Progress-Aware Video Frame Captioning

TL;DR

This paper introduces progress-aware video frame captioning, a new task that generates detailed, temporally precise descriptions for each video frame, capturing action progression and advancing video understanding.

Contribution

We propose ProgressCaptioner, a novel model for frame-level captioning, along with the FrameCap dataset and FrameCapEval benchmark, to improve temporal captioning accuracy.

Findings

ProgressCaptioner outperforms existing models in capturing action progression.

The FrameCap dataset enables effective training of fine-grained captioning models.

Our approach improves keyframe selection and overall video understanding.

Abstract

While image captioning provides isolated descriptions for individual images, and video captioning offers one single narrative for an entire video clip, our work explores an important middle ground: progress-aware video captioning at the frame level. This novel task aims to generate temporally fine-grained captions that not only accurately describe each frame but also capture the subtle progression of actions throughout a video sequence. Despite the strong capabilities of existing leading vision language models, they often struggle to discern the nuances of frame-wise differences. To address this, we propose ProgressCaptioner, a captioning model designed to capture the fine-grained temporal dynamics within an action sequence. Alongside, we develop the FrameCap dataset to support training and the FrameCapEval benchmark to assess caption quality. The results demonstrate that ProgressCaptioner significantly surpasses leading captioning models, producing precise captions that accurately capture action progression and set a new standard for temporal precision in video captioning. Finally, we showcase practical applications of our approach, specifically in aiding keyframe selection and advancing video understanding, highlighting its broad utility.