KFS-Bench: Comprehensive Evaluation of Key Frame Sampling in Long Video Understanding

TL;DR

KFS-Bench is a new benchmark for evaluating key frame sampling strategies in long video question answering, enabling direct assessment of sampling quality and proposing a novel method that improves scene coverage and QA accuracy.

Contribution

This work introduces KFS-Bench, the first benchmark with multi-scene annotations for direct evaluation of key frame sampling in long videos, and proposes a new sampling method leveraging question-video relevance.

Findings

Sampling precision, scene coverage, and sampling balance are key factors affecting QA performance.

A novel sampling quality metric correlates well with QA accuracy.

The proposed adaptive sampling method outperforms existing approaches.

Abstract

We propose KFS-Bench, the first benchmark for key frame sampling in long video question answering (QA), featuring multi-scene annotations to enable direct and robust evaluation of sampling strategies. Key frame sampling is crucial for efficient long-form video understanding. In long video QA, selecting informative frames enables multimodal large language models (MLLMs) to improve both accuracy and efficiency. KFS-Bench addresses the limitation of prior works that only indirectly assess frame selection quality via QA accuracy. By providing ground-truth annotations of multiple disjoint scenes required per question, KFS-Bench allows us to directly analyze how different sampling approaches capture essential content across an entire long video. Using KFS-Bench, we conduct a comprehensive study of key frame sampling methods and identify that not only sampling precision but also scene coverage and sampling balance are the key factors influencing QA performance. Regarding all the factors, we design a novel sampling quality metric that correlates with QA accuracy. Furthermore, we develop a novel key frame sampling method that leverages question-video relevance to balance sampling diversity against question-frame similarity, thereby improving coverage of relevant scenes. Our adaptively balanced sampling approach achieves superior performance in both key frame sampling and QA performance. The benchmark is available at https://github.com/NEC-VID/KFS-Bench.