Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMs

TL;DR

Q-Frame introduces an adaptive, query-aware frame selection method for Video-LLMs that enhances video comprehension by efficiently capturing crucial spatiotemporal information without increasing computational costs.

Contribution

The paper presents a training-free, plug-and-play approach for adaptive frame selection and multi-resolution scaling tailored to specific queries in video understanding.

Findings

Q-Frame outperforms existing methods on benchmark datasets.

It enables processing more frames without additional computational costs.

Demonstrates versatility across various video understanding tasks.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated significant success in visual understanding tasks. However, challenges persist in adapting these models for video comprehension due to the large volume of data and temporal complexity. Existing Video-LLMs using uniform frame sampling often struggle to capture the query-related crucial spatiotemporal clues of videos effectively. In this paper, we introduce Q-Frame, a novel approach for adaptive frame selection and multi-resolution scaling tailored to the video's content and the specific query. Q-Frame employs a training-free, plug-and-play strategy generated by a text-image matching network like CLIP, utilizing the Gumbel-Max trick for efficient frame selection. Q-Frame allows Video-LLMs to process more frames without exceeding computational limits, thereby preserving critical temporal and spatial information. We demonstrate Q-Frame's effectiveness through extensive experiments on benchmark datasets, including MLVU, LongVideoBench, and Video-MME, illustrating its superiority over existing methods and its applicability across various video understanding tasks.