GIFT: Global Irreplaceability Frame Targeting for Efficient Video Understanding

TL;DR

GIFT is a training-free frame selection method for video understanding that assesses frame irreplaceability to improve efficiency and accuracy, outperforming existing methods on long-form video benchmarks.

Contribution

It introduces a novel irreplaceability-based frame selection framework with Directed Diversity and Budget-Aware Refinement, addressing limitations of greedy approaches.

Findings

GIFT achieves up to 12.5% improvement over uniform sampling on LLaVA-Video-7B.

It effectively balances relevance and diversity in frame selection.

The method is training-free and adaptable to various video understanding tasks.

Abstract

Video Large Language Models (VLMs) have achieved remarkable success in video understanding, but the significant computational cost from processing dense frames severely limits their practical application. Existing methods alleviate this by selecting keyframes, but their greedy decision-making, combined with a decoupled evaluation of relevance and diversity, often falls into local optima and results in erroneously selecting irrelevant noise frames. To address these challenges, we propose GIFT: Global Irreplaceability Frame Targeting, a novel training-free framework that selects frames by assessing their intrinsic irreplaceability. Specifically, we first introduce Directed Diversity to quantify a frame's uniqueness conditioned on relevance, which allows us to formulate a unified irreplaceability score. Subsequently, our Budget-Aware Refinement strategy employs a adaptive iterative process that first secures a core set of frames with the highest irreplaceability, and then shifts its priority to building crucial temporal context around these selections as the budget expands. Extensive experiments demonstrate that GIFT achieves a maximum average improvement of 12.5% across long-form video benchmarks on LLaVA-Video-7B compared to uniform sampling.