Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenders

TL;DR

This paper introduces Nar-KFC, a modular approach that enhances long video understanding by selecting keyframes through optimization and supplementing with generated narratives, improving MLLMs' performance efficiently.

Contribution

It proposes a novel keyframe selection and narrative insertion method, addressing computational challenges and temporal discontinuity in long video comprehension with MLLMs.

Findings

Significant performance improvements on multiple benchmarks.

Efficient keyframe selection via greedy search.

Enhanced coherence with narrative insertion.

Abstract

Employing Multimodal Large Language Models (MLLMs) for long video understanding remains a challenging problem due to the dilemma between the substantial number of video frames (i.e., visual tokens) versus the limited context length of language models. Traditional uniform sampling often leads to selection of irrelevant content, while post-training MLLMs on thousands of frames imposes a substantial computational burden. In this paper, we propose threading keyframes with narratives (Nar-KFC), a plug-and-play module to facilitate effective and efficient long video perception. Nar-KFC generally involves two collaborative steps. First, we formulate the keyframe selection process as an integer quadratic programming problem, jointly optimizing query-relevance and frame-diversity. To avoid its computational complexity, a customized greedy search strategy is designed as an efficient alternative. Second, to mitigate the temporal discontinuity caused by sparse keyframe sampling, we further introduce interleaved textual narratives generated from non-keyframes using off-the-shelf captioners. These narratives are inserted between keyframes based on their true temporal order, forming a coherent and compact representation. Nar-KFC thus serves as a temporal- and content-aware compression strategy that complements visual and textual modalities. Experimental results on multiple long-video benchmarks demonstrate that Nar-KFC significantly improves the performance of popular MLLMs. Code will be made publicly available.