VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DR

TL;DR

VIBE is an annotation-free evaluation method for video summaries that improves decision-making by selecting summaries based on grounding and utility scores, enhancing task accuracy and efficiency.

Contribution

VIBE introduces a novel annotation-free framework for evaluating and selecting video summaries using grounding and utility metrics, improving downstream task performance.

Findings

Summaries selected by VIBE boost task accuracy by up to 61.23%.

VIBE reduces response time by up to 75.77%.

Effective human decision-making is supported without costly annotations.

Abstract

Many decision-making tasks, where both accuracy and efficiency matter, still require human supervision. For example, tasks like traffic officers reviewing hour-long dashcam footage or researchers screening conference videos can benefit from concise summaries that reduce cognitive load and save time. Yet current vision-language models (VLMs) often produce verbose, redundant outputs that hinder task performance. Existing video caption evaluation depends on costly human annotations and overlooks the summaries' utility in downstream tasks. We address these gaps with Video-to-text Information Bottleneck Evaluation (VIBE), an annotation-free method that scores VLM outputs using two metrics: grounding (how well the summary aligns with visual content) and utility (how informative it is for the task). VIBE selects from randomly sampled VLM outputs by ranking them according to the two scores to support effective human decision-making. Human studies on LearningPaper24, SUTD-TrafficQA, and LongVideoBench show that summaries selected by VIBE consistently improve performance-boosting task accuracy by up to 61.23% and reducing response time by 75.77% compared to naive VLM summaries or raw video.