Triage: Hierarchical Visual Budgeting for Efficient Video Reasoning in Vision-Language Models

TL;DR

Triage is a hierarchical, training-free framework that improves the efficiency of video reasoning in vision-language models by strategically allocating computational resources through visual budgeting, without sacrificing performance.

Contribution

It introduces a novel hierarchical visual budgeting approach for video reasoning that is training-free and plug-and-play, significantly enhancing efficiency while maintaining accuracy.

Findings

Increases inference speed and reduces memory usage.

Maintains or surpasses baseline performance on video reasoning benchmarks.

Effective hierarchical resource allocation improves efficiency in VLMs.

Abstract

Vision-Language Models (VLMs) face significant computational challenges in video processing due to massive data redundancy, which creates prohibitively long token sequences. To address this, we introduce Triage, a training-free, plug-and-play framework that reframes video reasoning as a resource allocation problem via hierarchical visual budgeting. Its first stage, Frame-Level Budgeting, identifies keyframes by evaluating their visual dynamics and relevance, generating a strategic prior based on their importance scores. Guided by this prior, the second stage, Token-Level Budgeting, allocates tokens in two phases: it first secures high-relevance Core Tokens, followed by diverse Context Tokens selected with an efficient batched Maximal Marginal Relevance (MMR) algorithm. Extensive experiments demonstrate that Triage improves inference speed and reduces memory footprint, while maintaining or surpassing the performance of baselines and other methods on various video reasoning benchmarks.