LFS: Learnable Frame Selector for Event-Aware and Temporally Diverse Video Captioning

TL;DR

This paper introduces LFS, a learnable frame selector that improves video captioning by selecting diverse, event-relevant frames using caption feedback, leading to better detailed descriptions and question answering.

Contribution

LFS explicitly models temporal importance and uses caption feedback from frozen LLMs to optimize frame selection for enhanced video captioning.

Findings

LFS improves caption quality by up to 2.0% on VDC and over 4% on ICH-CC benchmarks.

LFS enhances performance on video question answering tasks.

Introduces ICH-CC, a human-aligned video understanding benchmark.

Abstract

Video captioning models convert frames into visual tokens and generate descriptions with large language models (LLMs). Since encoding all frames is prohibitively expensive, uniform sampling is the default choice, but it enforces equal temporal coverage while ignoring the uneven events distribution. This motivates a Learnable Frame Selector (LFS) that selects temporally diverse and event-relevant frames. LFS explicitly models temporal importance to balance temporal diversity and event relevance, and employs a stratified strategy to ensure temporal coverage while avoiding clustering. Crucially, LFS leverages caption feedback from frozen video-LLMs to learn frame selection that directly optimizes downstream caption quality. Additionally, we identify the gap between existing benchmark and human's cognition. Thus, we introduce ICH-CC built from carefully designed questions by annotators that reflect human-consistent understanding of video. Experiments indicate that LFS consistently improves detailed video captioning across two representative community benchmarks and ICH-CC, achieving up to 2.0% gains on VDC and over 4% gains on ICH-CC. Moreover, we observe that enhanced captions with LFS leads to improved performance on video question answering. Overall, LFS provides an effective and easy-to-integrate solution for detailed video captioning.