Learning Question-Aware Keyframe Selection with Synthetic Supervision for Video Question Answering

TL;DR

This paper introduces a question-aware keyframe selection method for VideoQA that uses synthetic supervision and coverage regularization, significantly improving accuracy and reasoning efficiency.

Contribution

It proposes a novel framework combining pseudo labels from LMMs and coverage regularization to enhance keyframe selection in VideoQA.

Findings

Improved accuracy on NExT-QA, especially for temporal and causal questions.

Effective keyframe selection as a learnable module.

Enhanced reasoning efficiency in VideoQA.

Abstract

Large multimodal models (LMMs) have recently demonstrated remarkable performance in video question answering (VideoQA), yet reasoning over video remains challenging due to high inference cost and diluted information. Keyframe selection offers efficiency and sharper reasoning but suffers from sparse supervision and redundant frame choices when relying only on image-text similarity. We present a question-aware keyframe selection framework with two components: pseudo keyframe labels derived from LMMs that provide informative supervision and a coverage regularization that promotes diverse, complementary evidence across time. Experiments on NExT-QA show that our method significantly improves accuracy, especially for temporal and causal question types, establishing keyframe selection as an effective and learnable module for VideoQA.