Enhancing Long Video Question Answering with Scene-Localized Frame Grouping

TL;DR

This paper introduces SceneQA, a new long video question-answering scenario, along with the LVSQA dataset, and proposes SLFG, a scene-localized frame grouping method that improves multimodal large language models' understanding of long videos without altering their architecture.

Contribution

The paper presents SceneQA and LVSQA for evaluating scene-based reasoning in long videos, and introduces SLFG, a novel scene-localized frame grouping technique that enhances model performance.

Findings

SLFG significantly improves long video understanding in benchmarks.

LVSQA provides a fair evaluation of scene perception abilities.

The approach requires no modification to existing models.

Abstract

Current Multimodal Large Language Models (MLLMs) often perform poorly in long video understanding, primarily due to resource limitations that prevent them from processing all video frames and their associated information. Efficiently extracting relevant information becomes a challenging task. Existing frameworks and evaluation tasks focus on identifying specific frames containing core objects from a large number of irrelevant frames, which does not align with the practical needs of real-world applications. To address this issue, we propose a new scenario under the video question-answering task, SceneQA, which emphasizes scene-based detail perception and reasoning abilities. And we develop the LVSQA dataset to support the SceneQA task, which is built upon carefully selected videos from LVBench and contains a new collection of question-answer pairs to promote a more fair evaluation of MLLMs' scene perception abilities in long videos. Inspired by human cognition, we introduce a novel method called SLFG. The core idea of SLFG is to combine individual frames into semantically coherent scene frames. By leveraging scene localization methods and dynamic frame reassembly mechanisms, SLFG significantly enhances the understanding capabilities of existing MLLMs in long videos. SLFG requires no modification to the original model architecture and boasts excellent plug-and-play usability. Experimental results show that this method performs exceptionally well in several long video benchmark tests. Code and dataset will be released at http://www.slfg.pkuzwh.cn.