Moment Sampling in Video LLMs for Long-Form Video QA

TL;DR

This paper introduces 'moment sampling', a novel, model-agnostic method that improves long-form VideoQA by selecting the most relevant frames based on question context, enhancing reasoning and efficiency.

Contribution

We propose a general-purpose, lightweight moment retrieval-guided frame sampling method to better select relevant frames for long-form VideoQA tasks.

Findings

Improves accuracy on four long-form VideoQA datasets

Reduces redundant frame processing and computational costs

Enhances reasoning capabilities in Video LLMs

Abstract

Recent advancements in video large language models (Video LLMs) have significantly advanced the field of video question answering (VideoQA). While existing methods perform well on short videos, they often struggle with long-range reasoning in longer videos. To scale Video LLMs for longer video content, frame sub-sampling (selecting frames at regular intervals) is commonly used. However, this approach is suboptimal, often leading to the loss of crucial frames or the inclusion of redundant information from multiple similar frames. Missing key frames impairs the model's ability to answer questions accurately, while redundant frames lead the model to focus on irrelevant video segments and increase computational resource consumption. In this paper, we investigate the use of a general-purpose text-to-video moment retrieval model to guide the frame sampling process. We propose "moment sampling", a novel, model-agnostic approach that enables the model to select the most relevant frames according to the context of the question. Specifically, we employ a lightweight moment retrieval model to prioritize frame selection. By focusing on the frames most pertinent to the given question, our method enhances long-form VideoQA performance in Video LLMs. Through extensive experiments on four long-form VideoQA datasets, using four state-of-the-art Video LLMs, we demonstrate the effectiveness of the proposed approach.